for uni im going to be coding with a chatGPT user, so i decided to see how good it is at coding (sure ive heard it can code, but theres a massive difference between being able to code and being able to code well).
i will complain about a specific project i asked it to make and improve on under the cut, but i will copy my conclusion from the bottom of the post and paste it up here.
-
conclusion: it (mostly) writes code that works, but isnt great. but this is actually a pretty big problem imo. as more and more people are using this to learn how to code, or getting examples of functions, theyre going to be learning from pretty bad code. and then theres what im going to be experiencing, coding with someone who uses this tool. theres going to be easily improvable code that the quote unquote writer wont fully understand going into a codebase with my name of it - a codebase which we will need present for our degree. even though the code is not the main part of this project (well, the quality of the code at least. you need it to be able to run and thats about it) its still a shitty feeling having my name attached to code of this quality.
and also it is possible to get it to write good (readable, idiomatic, efficient enough) code, but only if you can write this code yourself (and are willing to spend more time arguing with the AI than you would writing the code.) most of the things i pointed out to the AI was stuff that someone using this as a learning resource wont know about. if it never gives you static methods, class methods, ABCs, coroutines, type hints, multi-file programs, etc without you explicitly asking for them then its use is limited at best. and people who think that its a tool that can take all the info they need, and give it back to them in a concise, readable way (which is a surprising lot of people) will be missing out without even knowing about it.
i got it to write tic-tac-toe (the standard babee) in python (the lang i have to use for uni ;-; (held at gunpoint here)). my specific prompt was "write me a python program for tictactoe that is written in an object oriented way and allows for future expansion via multiple files"
it separated it into three files below (which i think would run, but i never actually ran any of this code. just reading and judging)
why does board use display instead of __str__ and __repr__?
why is the board stored as 1d instead of 2d? thats just confusing
why does it never early return aside from check_winner? (not a big issue here but kept on choosing to never early return when i asked it to add more methods)
why is there no handling of non-number user inputs?
why are non-int inputs truncated instead of telling the user that they should input ints only?
why is display implemented like that?
why are so many lines so bloody long (wide)?
why is there a redundant self.check_winner() after the while loop in TicTaacToe.play()? and if it wasnt redundant then you could finish the game without there being anything printed telling you that the game is finished?
why is the only comment useless? (this comment wouldnt be useless if it was a doc comment tho, but it aint a doc comment. speaking of, why is there no doc comments?)
these are the more immediate things i saw, but there are other things that are bad here.
whenever i write * this is where it updated the api without changing any usage of the api.
so i ask it to change board.display into __str__ and __repr__, it changes it to __str__*, it does not add a __repr__. asking it to add a __repr__ 1) removes the __str__ and 2) gives me this (the other methods are unchanged)
what. the. fuck. this would imply that board takes in an argument for the boardstate, but it clearly doesnt. after 4 more asks it finally has both __str__ and __repr__, without fixing the fact its implying board takes an optional arg, so i get it to add this arg. anything that needs to print the board still calls display btw.
the reasoning it gave for using display over the repr and str magics was this
While using __str__ and __repr__ is a more idiomatic approach in Python, choosing to have a separate display method can still be a valid choice, especially if the display logic is more complex or if you want to keep the __str__ method for a more concise or formal representation of the object.
which, erm what? why would __str__ be for a concise or formal repr when thats what __repr__ is for? who cares about how complex the logic is. youre calling this every time you print, so move the logic into __str__. it makes no difference for the performance of the program (if you had a very expensive func that prints smth, and you dont want it to run every time you try to print the obj then its understandable to implement that alongside str and repr)
it also said the difference between __str__ and __repr__ every damn time, which if youre asking it to implement these magics then surely you already know the difference?
but okay, one issue down and that took what? 5-10 minutes? and it wouldve taken 1 minute tops to do it yourself?
okay next implementing a tic-tac-toe board as a 1d array is fine, but kinda weird when 2d arrays exist. this one is just personal preference though so i got it to change it to a 2d list*. it changed the init method to this
tumblr wont let me add alt text to this image so:
[begin ID: Python code that generates a 2D array using nested list comprehensions. end ID]
which works, but just use [[" "] * 3 for _ in range(3)]. the only advantage listcomps have here over multiplying is that they create new lists, instead of copying the pointers. but if you update a cell it will change that pointer. you only need listcomps for the outermost level.
again, this is mainly personal preference, nothing major. but it does show that chatgpt gives u sloppy code
(also if you notice it got rid of the board argument lol)
now i had to explicitly get it to change is_full and make_move. methods in the same damn class that would be changed by changing to a 2d array. this sorta shit should be done automatically lol
it changed make_move by taking row and col args, which is a shitty decision coz it asks for a pos 1-9, so anything that calls make_move would have to change this to a row and col. so i got it to make a func thatll do this for the board class
what i was hoping for: a static method that is called inside make_move
what i got: a standalone function that is not inside any class that isnt early exited
the fuck is this supposed to do if its never called?
so i had to tell it to put it in the class as a static method, and get it to call it. i had to tell it to call this function holy hell
like what is this?
i cant believe it wrote this method without ever calling it!
and - AND - theres this code here that WILL run when this file is imported
which, errrr, this files entire point is being imported innit. if youre going to have example usage check if __name__ = "__main__" and dont store vars as globals
now i finally asked it to update the other classes not that the api has changed (hoping it would change the implementation of make_move to use the static method.) (it didnt.)
Player.make_move is now defined recursively in a way that doesnt work. yippe! why not propagate the error ill never know.
also why is there so much shit in the try block? its not clear which part needs to be error checked and it also makes the prints go offscreen.
after getting it to fix the static method not being called, and the try block being overcrowded (not getting it to propagate the error yet) i got it to add type hints (if u coding python, add type hints. please. itll make me happy)
now for the next 5 asks it changed 0 code. nothing at all. regardless of what i asked it to do. fucks sake.
also look at this type hint
what
the
hell
is
this
?
why is it Optional[str]???????? the hell??? at no point is it anything but a char. either write it as Optional[list[list[char]]] or Optional[list[list]], either works fine. just - dont bloody do this
also does anything look wrong with this type hint?
a bloody optional when its not optional
so i got it to remove this optional. it sure as hell got rid of optional
it sure as hell got rid of optional
now i was just trying to make board.py more readable. its been maybe half an hour at this point? i just want to move on.
it did not want to write PEP 8 code, but oh well. fuck it we ball, its not like it again decided to stop changing any code
(i lied)
but anyway one file down two to go, they were more of the same so i eventually gave up (i wont say each and every issue i had with the code. you get the gist. yes a lot of it didnt work)
conclusion: as you probably saw, it (mostly) writes code that works, but isnt great. but this is actually a pretty big problem imo. as more and more people are using this to learn how to code, or getting examples of functions, theyre going to be learning from pretty bad code. and then theres what im going to be experiencing, coding with someone who uses this tool. theres going to be easily improvable code that the quote unquote writer wont fully understand going into a codebase with my name of it - a codebase which we will need present for our degree. even though the code is not the main part of this project (well, the quality of the code at least. you need it to be able to run and thats about it) its still a shitty feeling having my name attached to code of this quality.
and also it is possible to get it to write good (readable, idiomatic, efficient enough) code, but only if you can write this code yourself (and are willing to spend more time arguing with the AI than you would writing the code.) most of the things i pointed out to the AI was stuff that someone using this as a learning resource wont know about. if it never gives you static methods, class methods, ABCs, coroutines, type hints, multi-file programs, etc without you explicitly asking for them then its use is limited at best. and people who think that its a tool that can take all the info they need, and give it back to them in a concise, readable way (which is a surprising lot of people) will be missing out without even knowing about it.
