People really need to lose the idea that there's any Intelligence going on in GPT etc, or that it knows anything or gives answers to questions. It gives the illusion of those things, by algorithmically chaining words together in an order which looks like what you want.
And the unwary will get bitten.
I've encountered several people through work now who have rejected what I - a person who has been in this field for THIRTY YEARS - tell them because their text generator of choice told them something different. Even when the reason they called was that what the machine told them to do didn't work.
Yes, sure, the thing has probably slurped in every single forum post even vaguely related to that thing which has ever been made, and I haven't, but all it's doing is putting out something which looks like answers someone on a forum would give. I KNOW things and understand them, and if I don't, I can take in new information and reason and come to an actual answer.
A text generator doesn't understand the topic, doesn't know how to select which sources are relevant and which aren't, probably doesn't make any value judgements on the sources it's using (and even if it is, it won't be using the same criteria a person would use, especially not on a subject that person has a reasonable understanding of already) and it doesn't and can't check that what it's saying is right, because it has no concept of right or wrong or even that what it's outputting has any meaning, all it can do is use algorithms to determine what a text of a particular type is likely to have in it.
It is, in short, designed to fool you.
And that's no different when what it's outputting is a command or code snippet. I've seen these things give PowerShell commands with options which don't exist for the cmdlet in question, and which will therefore error, but what's worse is when the code doesn't error, but also doesn't do what the user wanted. They're getting the wrong result but they don't know that because they don't know what any of the code actually does or how to check anything. And, increasingly, it seems that they don't even realise they ought to be checking.
One of the calls I had was from someone who was trying to do something in a particular programming language and were using three different development environments to edit various parts of their code. They were asking chatGPT how to do everything and it must have been pulling from multiple environment-specific forums.
Their latest generated instructions told them to run commands which checked the version of the programming language and if those commands said it wasn't found to install it, so they called to get me to do that as it required admin credentials.
I just happened to have started looking at that language myself a few days before. I hadn't got any further than installing it, but I knew it was asking for an old version and so queried what they were actually trying to achieve.
They refused to explain the over-arching goal but said they wanted to use a particular API. A couple of minutes reading the dev site for that API told me it had been retired in 2018 and, when I finally got more information about what they were doing in an attempt to find an alternative, I realised it wouldn't have done quite what they wanted even when it was current.
I showed them that, advised that they could do all of the code in one environment and opened the relevant files in the least resource-hungry one (they'd also had issues with the machine running out of memory), pointed them at the dev documentation and also found them a thread on a forum which covered doing what they actually wanted to. Because I had other people to help and wasn't going to do their job for them.
They seemed satisfied. But before I disconnected from their machine, I saw them closing the browser tabs I'd opened and going back to chatGPT. Not long after that, my boss called because they'd emailed him and their own boss, complaining about me "refusing to help" and saying that I clearly didn't know what I was doing, shouldn't waste their time pretending I did and it was ridiculous to suggest they waste more time reading the "nonsense" I'd pointed them at. My boss had read the notes I'd put on the ticket which detailed what I'd seen and done, he just wanted to check he'd understood.
"Just so we're clear," he said. "You've never used this language, you just spent a couple of hours at the weekend getting to the point of being able to build the abandoned git project you wanted to fix a bug in, and then you spent half an hour on the phone with them, and apparently ended up with a better understanding than they have of how to do what they've spent the last three months trying to do."
"And probably getting paid more than me to do it, yes. Don't worry, I don't monetize my hobbies."
A couple of weeks later I processed a leaver request for them. I assume those things were related.
Asking one of these things might seem like it's a quicker way to get things done, and if you get lucky and get the right answers then yeah it probably lets you get started quicker. But it's only ever going to get you so far.
If you don't spend the time actually learning how to do things, don't make the effort to understand, then you're going to waste a lot of time asking and trying and failing and then asking again and never actually understanding enough to do anything without asking, because what you're being given is all small, disjointed pieces which don't link up enough for you to get a picture of the underlying structure. And without that, you can't figure anything out for yourself. You'll never get to the point where you can just sit down and make a complete thing, you'll always be spending more time querying than doing. You'll never be any faster than a beginner.
It's the opposite of what it promises to do for you.
