SceneMax3D Development

About the development of the SceneMax3D programming language and related components

[adi-barda]

SceneMax3D is Going Open Source

I asked Claude - does SceneMax3D has sufficient quality to go open source? I don't want to be ashamed in front of the community... :) Claude analyzed the entire project, suggested some critical security changes. We decided to upload the code in a new initial commit without the entire ~10 years of coding history. So... here it is: https://github.com/scenemax3d/scenemax3d-desktop/tree/main

We have so many game engines. Why the hell do we need another one? what makes SceneMax3D unique to attract the community attention? Well it was built for educational purposes in mind and it has built-in virtual classrooms support so that anyone can teach from remote how to create games, accept and resolve questions in real-time everything from within the IDE. This is the strongest differentiator. It also has it own programming language which is kind of unique although Godot also has its own GDScript language. I plan to add an extensibility mechanism to anyone who wants to inject Java code (App states middleware) for maximum performance but the main intention is that people will use the scripting language for their games because it is so easy.

Interesting times a head...

[adi-barda]

War-time Coder

On Saturday Feb - 28 ~08:00 am, I was riding my Yamaha Tenere 700 in the beautiful area of Tal-Shahar when suddenly a group of jet fighters flew over my head. Immediately I understood that war is starting (again). Two minutes later the sirens began. I turned the bike and drove home.

my beloved Yamaha Tenere 700 World Raid US-Israel vs Iran war has started with very intensive fire. So here is a description of a programmer life during war time - 1. You work 100% from home. The offices are officially open but it's very dangerous to drive to Tel Aviv since it's the major target for the Iranians. 2. There's an average of 10 alerts a day in my city Petah Tikva. Your cell phone becomes the most important tool. It's a life saving tool that needs to be charged and available at all times. An alert event is a composition of multiple alerts: * Long range alert - A missile (s) is detected fired from Iran and it is likely to hit your city - you get the first part of the alert in your phone - very strong and scary sound - it means that you should prepare for a missile hitting in 5 to 10 minutes and you need to be near a shelter. In the first 3-5 minutes, missile interceptors in foreign countries as well as layer 1 - Israel's "Arrow 3" interceptor tries to hit the coming missile. During this time you can hardly function at work.

Long range alert - you need to be close to a shelter * Short range alert - In case layer 1 interception doesn't succeed you get the second part of the alert in your phone, one and a half minutes before the hit. This time you need to hurry and go inside a shelter, lock everything and stay there for at least 10 minutes until you get the ending alert. During those 12 minutes you cannot function at all at work. * Ending alert - the event is closed and you may go out of the shelter and continue life as usual.

Home front application - alerts when need to find a shelter Each of those alerts triggered twice so a total of 6 alerts per missile event. So far we had maximum of 15 events per day and a minimum of 3 events. This is best case scenario when a missile is launched from Iran. If it comes from Lebanon then you don't have the long range alert. If Iran is coordinating the shooting with Lebanon then you get multiple event at the same time (many many alerts...). If short range alert is triggered then you almost every time going to hear a significant BOOM either because the missile has landed or it was intercepted but the missile's debris is landing (very scary as well).

In the past 18 days - 108 actual alerts (where you need to run to a shelter) in my town Petah Tikva Worst case scenario - missile has landed and hit a civilian building. In that case there will be casualties and a lot of damage. Personally I have an apartment which was totally lost in the previous, "12 days war" with Iran - the entire building has collapsed. Luckily the resident ran out to a public shelter on time and was saved. So, can you work as a coder during war time? short answer is yes you can. Anyone who works in a global corporate need to deliver no matter what. Long answer is that it's complicated - your sleep is not good, sometimes you need to run to a shelter 3-4 times a night. You are always in some kind of tension. If you have kids you need to function as a teacher (because there is no school) and you also need to teach them what to do in case of emergency when they are not at home - if the missile is coming from Lebanon then to hide behind the south side of a wall. If it's coming from Iran, hide behind west side and if it's from Gaza then seek for the east side and of course social media is full of horror clips and stories which needs mitigation. Sprint planning at work - sprints are the same 9 working days but we all know it's much less because of the situation so pressure rise up. As strange as it may sound - Israelis respect and like the Iranians and Lebanese and we dream on the day we will have peace with them. This day is coming very soon - we know that.

[adi-barda]

(Yet another) Thoughts on AI

For us programmers, using AI for doing our job is like using sever drags - it's addicted, you feel amazing, you deliver features in speedlight your productivity is skyrocketing but then after a short period of few month you realize that you are no longer a programmer and it hurts. The entire industry is going in that direction. Programmers are moving fast towards SDD (Spec Driven Development). You are getting expensive annual AI license and you are expected to use it. You can write your own code but you are expected to start and maintain features using SDD. In the beginning you review every line of code that the AI generates and as the time goes on, your trust level of the generated code becomes so high that you are mostly doing functional sanity testing. Just in the last few weeks AI tools such as Claude code, Gemini, ChatGPT and Lovable helped me to deliver those products: 1. A full website from scratch for reviewing offroad Israeli tracks. I'm not a UI programmer and Lovable did all the UI for me. it did it exceptionally well and met every requirement I gave it.

https://www.advrider.co.il 2. Added Android-Auto screens to my ADV Rally Android offroad navigation application, A project which I thought will take me 8 month or so was completed in 3 weeks passing all Google's tests and delivered to production giving my offroad community amazing value.

https://play.google.com/store/apps/details?id=com.abware.watchdog_client 3. An MCP server for my wife's business database allowing her to ask any complicated questions about her patients, her treatments, recommendations etc. saving her tons of time. 4. Offline maps generation - refactoring my old Lua scripts and MapBox style files so now I can produce much better maps for my offroad community. 5. SceneMax3D IDE - improved my games development IDE's look and feel as well as editing user experience.

6. SceneMax3D engine performance - I asked Claude to analyze the performance of the language's interpreter and it did great job finding Java optimizations that I can use. I asked it to implement it's suggestions and this time I had to carefully review every single line of code and do thorough testing to make sure that no regressions were introduced. Claude did an amazing job. 7. Fixed every gradle hell that I had in my Java projects including updating gradle, updating dependencies fixing build errors and compiling the projects until everything worked. This task alone could take me weeks to solve. 8. I even asked Claude to help me re-writing some of my Fighting game code which is written in SceneMax3D language so Claude has no previous knowledge about programming with it (!) and about design patterns using that language and even so - it did an amazing job.

Warrior - Fighting Game Suddenly context switching between projects is as easy as switching the project folder in Claude's UI. No more hours of struggling to switch to a project that you have not visited for a long time. So, productivity wise you become a super developer. The problem is that you are no longer a software programmer, not a software engineer, not a coder. You become something else.

So what will happen to us? All of us will loose our jobs. We will have other job but not software engineering.

Until today we had an idea, a vision and we had the knowledge , the talent and the mental strength (and tons of coffee) to translate those ideas to working machine code. We spent years and years to learn the machine coding. Generally the coding part is solved - the machine has learnt our human language and we can just "talk" to each other and it will do the coding part for us. I feel petty for the younger generation of software engineers. Soon they will stop coding at all and will switch to 100% writing specs. Not even code reviews, no testing - nothing. I feel so lucky that in the past 25 years or so I programmed all my projects by my own. I had the privilege to do actual coding. Younger generation of software engineers will not have that experience. The industry will not let them because it's counter productive. Actual coding will be used as a hobby for those who wants to have a deeper understanding of software. People will still invest time and money to learn how to play the piano even though AI can create any piano ballad they wish, because it has value to learn and become a better version of yourself. I guess the same goes for learning how to code. Now, that the coding part is solved, production bottlenecks will switch to product personnel, functional architects, product managers, customer success, sales and marketing.

[adi-barda]

Sports

Generally speaking, I'm not interested in sports. I find most of it to be extremely boring to watch and hard to practice. Funny enough my first degree was in physical education with specialization in seamanship (sailing?). I used to teach windsurfing for a couple of years, was a Gym instructor in the University but again general sports doesn't talk to me.

Me trying to do pull-ups...

Except for one - martial arts. This is the one sports which I find interesting. I learned Ju-Jitsu for 14 years, Denis survival (3 years), then Shotokan Karate (4 years). Then I took a break for 13 years! because of personal issues and a few month ago I decided to gradually come back to practicing martial arts, but which one will I take? Ju-Jitsu and Karate are not possible for me because of past injuries. I tried Kung-Fu it was nice but I couldn't kick due to those injuries. I needed something which uses mainly the body and hands, nothing else and I found it - classic Boxing. I thought "Oh what a wonderful easy sport! just using your hands, no kicking..." and Oh was I wrong about it. It is everything but easy. Yes there are no kicks but your legs are constantly working and keeping your hands up for an hour with those heavy gloves - it's very hard. In Karate for example you can allow your hands to rest for a few seconds by using your legs instead. Same goes for Ju-Jitsu, Taekwondo, Kung-Fu etc. So classic boxing is definitely not easy as I thought but I like it. It's smart, has a glorious past, my coach is phenomenal very professional and the guys are really nice.

[adi-barda]

Unity 3D Short Review

* this text is 100% human. No AI is involved in generating it In the last year I've been using Unity for my studies in game development. I was able to create and publish 3 small games in quite a short time. I was totally new to Unity, never used it before but I knew that it is one of the most popular game engine on the market.

Night Dancer - A game I've created in Unity in about 3 weeks in my spare time. The game includes video rendering, procedural generation and constant shape changes of the dancing stage in run-time, user input with hand gestures using open source machine learning model for the gesture recognition, AWS dynamo DB integration for presenting a Leader Board and the usual stuff - UI, particle system, Humanoid animation, events, triggers etc. Actually I never programed a game before in any other game engine except for my own SceneMax3D engine which is based on JME3 engine and a custom made programming language. So this is my short review for what I liked and what I didn't like about Unity3D. I consider myself as a Unity beginner level programmer so in that perspective you should read this text.

The good things I'm not going to write about each and every good feature in Unity (there's a lot of them) rather I would like to focus some good general components and concepts. For the most part, I think it is a fantastic game engine, very comprehensive with an amazing assets store and eco system. If I will ever aim for developing a serious full scale game, Unity probably will be my choice for game engine to be used.

Performance - this engine has great performance. I used a lot of scene objects, expensive particle systems, video rendering, just-in-time hundreds of objects (prefabs) loading and creation, music and sound effects and the engine handled them exceptionally good performance wise. Most of the time I don't even think about the performance cost of my game objects - that good is the engine's optimizations. Of course from time to time I do hit the performance limits and the FPS is dropping, then I apply some coding best practices and usually the problem is solved.

Slider Glider - a fast arcade game where you need to slide and jump on monstrous sliders while navigating your way in a procedurally generated maze. It took me about 3 weeks to develop this game in my spare time. The sliders models are placed in run-time and the maze is randomly generated in the beginning of each stage. There is also a simple navigation system assisting you to cope with the complex maze navigation reaching the treasure. The big challenge here was using curves for the sliding movement. Unity curves editor was very helpful in this case. Ease of use - Unity IDE is superb. The technique of dragging a prefab (pre defined composite of objects) onto a public field of some other game object and having it ready to be used at run time is very easy and intuitive to use. The usage of Mono Behaviors for injecting specific logic, the animation state machine, events and triggers are all contributing to the ease of using this engine. Rendering - Unity uses the best graphics APIs available for each targeted platform such as Vulkan and DirectX12. This affects the quality, portability and performance of your project.

Assets store - This one is by far my favorite feature of Unity. Absolutely amazing assets store with every library/graphic asset you can think of for your game and with superb reliable integration to your project for installing / uninstalling it. Some of the assets costs money and some of them are free. I was able to get anything I needed for free but for serious projects I will not hesitate to spend some money for getting state of the art libraries / assets. Build system - Unity supports building your projects for all major platforms including mobile, game consoles, PCs and web. This portability in my opinion is a big advantage Coding - For those of us (myself included) who prefers to build the scene graph using coding as much as possible in run time, Unity allows that freedom leaving you only the need to setup your game's resources (pre-fabs).

Dark Chaos - my first ever Unity game. Made in 3 weeks with absolutely no prior knowledge. Tons of particle systems, no best practices used, 3D models from Mixamo, Sketchfab and some I assembled by myself using Asset Forge. I think that demonstrates the ease of creating your first game with Unity Generative AI - Unity is probably the most popular game engine in the market using one the most popular general purpose programming language available (C#) so it makes it really easy for Vibe programing - letting an LLM to create some of your coding and code reviews for you. Harnessing Gen AI for writing some of your game's code is truly a revolution in game creation. Really hard tasks which might take days or even weeks for research and implementation is reduced to minutes. Let me give you an example - in my Night Dancer game I wanted a 10 by 10 dancing floor which changes its shape in real-time from "O" shape to "X" to "Love" to "Square" etc. in random sizes at random speed. That would take me quite a bit of time to code myself and it was perfectly done by the Gen AI in just a few minutes. Absolutely amazing. The bad things

As I mentioned earlier, I consider myself as a beginner programmer in Unity and also don't have much experience in any other game engines so the "bad things" that I'm going to list here might not be that bad if I had a wider perspective. External code editor - I find it really annoying that you can't code inside Unity IDE and need to switch back and forth from Unity IDE to an external code editor such as MS Visual Studio or VS Code. Each time you switch from one IDE to another, a compilation process starts followed by a synchronization process. It take a few seconds (sometimes quite a few!) and you need to wait for those processes to end before switching IDEs. Not fun especially for developers like myself who spend a lot of their time coding. UI Canvas - I find it a little bit confusing to arrange my UI components (2D) in the game's 3D world IDE freezes - It happened to me a few times that the IDE froze for some reason and sometimes the build process was stuck forever. I suspect that it is related to the anti virus installed on my Windows machine. Anyway I needed to hard stop Unity's process and restart the system risking some state loss

Summarize

I was really impressed with what Unity offers you out of the box and as an extension from its assets store and I highly recommend using this game engine for any serious game you have in mind. it is guaranteed that the engine will not be the one stopping you from completing your game. As for myself I'll probably be back working on simple games using my SceneMax3D and JME3 because this what I enjoy most - creating tools and software engineering. Cheers!

[adi-barda]

Back To Business

On Aug-12 2024 I have submitted a project for a global AI coding competition, Then for ~2 month I didn't (couldn't) touch any of my personal projects. I guess that's what happens when you "overflow" putting too much effort in 1 area and neglecting other areas. At work, I have accepted the challenge of becoming cloud security CIEM team leader. I'm excited for this new role but from past experience as a team leader in previous work place, I know that it comes with a lot of pressure. In addition, these days I'm starting my master degree in Digital Games Design. I scheduled my courses for the coming two semesters and (as expected) it consumes almost all of my free time. It also costs a lot of money but I feel that I owe it to myself to keep learning and developing my knowledge and meeting new people and having fun. I will not be able to finish it in 2 years so it is going to be a long 3 years journey... SceneMax3D might be involved in an interesting secret project. I can't tell much and I will not be doing the POC myself since I don't have any spare time but I will probably guide other developers how to use the engine. That should be a lot of fun to do :) What else - I take my time to learn the Irish Tin Whistle, It's such a fun addictive and practical piece of musical instrument. I highly recommend trying it out. It's not that easy to learn tough... Here is me trying to perform "The Foggy Dew" Irish war song:

Cheers!

[adi-barda]

Chapter 5 - Gemini API Developer Competition - Project Submission

The last two week were devoted to finalization of the AI Game Maker project: 1. Added attack strategy to the opponent in a fighting game - this feature allows the Gemini AI to assemble a fight strategy according to some statistics provided by the current game state. This is big topic by itself involving a lot of fine-tuning and cost optimizations (you don't want to use the AI to often while the game is running as it might cost quite a lot) 2. Prepare a new SceneMax3D installation with the new plugins system and upload it to the SceneMax3D website 3. Push the latest SceneMax3D mobile engine to the public repo in GitHub. The mobile engine is an open source project and it's 99% similar to the PC engine. 4. Prepare 2 sound tracks for the demo video clip. I used Suno AI for that and it was a very good experience. I highly recommend it for anyone who seeks to create amazing music for his products. 5. Prepare a short 3 minutes clip for the competition demo. I used Davinci Resolve for this task. I absolutely love this software and it's well worth the ~300$ I paid for it. The free version is also very good 6. Prepare Zip file with all the source code of this AI Game Maker project + binaries + instructions. This is a mandatory delivery for the competition.

7. Submission - on Aug-02-2024 I submitted my innovation. The last day for submission is on Aug-12 and then starts the fun part of reviewing other projects, voting etc. In about two months I start my master's degree in computer game design and the thesis is planned to be on the subject of AI generated games so a big part of that project is already made while participating in the Gemini competition - a win-win :)

My vision is to embed AI in major parts of SceneMax3D language and IDE. It opens a big opportunity for innovation.

Cheers!

[adi-barda]

Chapter 4 - Gemini API Developer Competition - Fighting game & Android Export

As planned, I spent the last days on adding fighting game capability to the engine and Android exporting feature. The fighting game has much more details in the puzzle for the AI agent to cope with. There are complex animations for the player and for the opponent, they need to constantly look at each other, you need to be able to demo their kick, punch, block animations, the player needs to be able to move in 3D space etc. Overall I'm very pleased with the results so far. The user can speak freely enough with the AI, get instant results and funny reactions. What's more, I've been able to add Android exporting of the game and automatically open it in Android studio. It was challenging because the Java code worked different on PC and on the mobile device specifically handling of Zip files and all kind of Gradle dependency hell. ChatGPT was on my side all the way, assisting me to resolve configuration issues and coding problems such as selecting the best Zip 3rd party library.

This video clip, demonstrates the current status of the project. It shows a complete story from the user perspective - you have a conversation with the AI, a game is created and finally you export it to Android studio for deployment in Google play store or any other market place.

What's next

Better and shorter presentation

Prepare the installation of all the components as well as SceneMax3D dev studio

Get feedback from the community

Prepare documentation for the architectural strategies, entities diagram etc.

So far I'm getting very good vibes from the game dev. community, and friends on various WhatsApp groups.

[adi-barda]

Chapter 3 - Gemini API Developer Competition - The importance of the unimportant input

The system is advancing as planned. I invest every minute of my spare time before and after work to push this project to the finish line.

So far I have a kind of complete story to show - The user starts the session with the computer, during that time he can freely speak to the Gemini agent and a game is developed in front of his eyes according to his wishes.

As mentioned in previous chapters, the role of Gemini AI is not to produce game code but rather to understand what the user wants up to a certain level of resolution so if we take a typical user sentence for example: "Hi Gemini, How are you? I would like to create a car racing game with a fast car and a professional racetrack. Can you help me please?" The Gemini prompt instruct the AI to dig just the "important" things, in this example - the user wants: 1. To create a car racing game 2. A professional race track 3. A fast car

The "unimportant" parts are: 1. Hi Gemini 2. How are you 3. Can you help me 4. Please I decided to response also to the "unimportant" parts so that the agent will react as fast as possible to user input such as "Thank you", "Hi", "Great" etc. The response can be an Emoji drawn to the game's screen along with a text and even some speech. This greatly helps the user to feel like he is having a session with a human being rather than a machine. My wife, which tested the system, liked this feature even more than the game creation itself!

What's next

1. Add support for fighting game creation 2. Allow exporting to Android native application

[adi-barda]

Chapter 2 - Gemini API Developer Competition

The last few days were intense. I needed to have a complete flow running with consistent results. Last time I created the plug-ins system for the SceneMax3D platform and an in-process webserver plugin for accepting commands from the outside world into the running game and modify it on-the-fly. This time, I was focused on creating a set of plugins for the SceneMax3D IDE as follows:

Speech To Text - I decided to use AssemblyAI as my speech to text provider. They provide an extremely easy to use speech to text Java SDK and registration process along with some nice features and a reasonable pricing model. For the speech to text feature, I decided to use the real-time streaming model. It was the coolest and easiest to implement.

Prompt Manager - the text provided by the speech-to-text plugin was handed to a prompt manager component which produced a well constructed and formatted prompt for Gemini AI including system instruction and user prompt.

Gemini AI API - the after mentioned prompt was delivered to a Gemini API component which is responsible for sending it to the Gemini AI service and return its result as plain text.

Configuration to code converter - the Gemini AI JSON text result is passed to a configuration to code converter which reproduces a valid SceneMax3D code. The code converter component has a built-in specific code converters for all kind of game genres such as racing games, fighting games, empty unknown games etc.

Gemini SceneMax3D Integration - this component is responsible for managing the entire workflow, orchestrating the various components described above from the user triggering the speech recognition and up to the generated SceneMax3D code which is sent to the run-time engine in real-time.

What's next

Now, that we have a complete working flow from the user interaction to the run-time game engine, we need to add the following features:

Enhance to configuration to code converter - support as much as possible user requests scenarios from updating a running scene up to switching to a new game concept while supporting as much as possible different game types.

Enhance the "Wow Factor" - make it look as impressive as possible emphasizing the Gemini AI contribution for the success of this process.

Export to Android devices - this is a requirement for this competition to have a Gemini AI product running on Android devices

Create a 3 minutes video clip demonstrating the product's capabilities.

[adi-barda]

Gemini API Developer Competition

So, I decided to compete in Google's AI developer competition. It's about writing an innovative application using the Gemini AI to perform some cool logic, submit a video clip explaining your innovation along with the source code and there's a chance of winning cool prizes. It's also exciting to test your innovation & coding skills. I decided to write a kind of real-time, AI based game creator - the user will talk to the AI agent (Gemini in this case), explain how the scene should look like and behave and according to that chat a game will be created and evolve in real-time. I already have built my own a game engine, SceneMax3D which uses the open source Java project JMonkey Engine for the scene rendering and other open source libraries such as Minie for the physics simulation etc. SceneMax3D has its own dedicated programming language and IDE which makes the AI developer competition project so much simpler to implement since I have control on all the components used for creating a 3D game and all that's left is to integrate Gemini AI API for creating a game…

So what is the initial plan?

Plugins system - Google requires the application's source code and there's no way to submit the entire SceneMax3D project sources so I decided to create a plug-ins mechanism for SceneMax3D and submit the Gemini integration plug-ins source code.

Language grammar - SceneMax3D programming language should support using any of the installed plugins so a new plugins command was added to the language. In this case, I need to activate an in-process web server while the engine is running. This will allow direct communication from the outside world to the engine while the game is running so after Gemini AI will generate its product, it could be streamed through a REST API using this in-proc web server.

Speech to text - the end-user should be able to express his thoughts by talking to the agent. The user's speech will be converted to text, which will be used inside some clever prompt, which will be sent to Gemini AI to do his magic.

IDE plugin - in addition to writing a plugin for the run-time engine, we will need one more plugin for the IDE. Its role is to accept the Gemini product, convert it to some SceneMax3D code and send that code to the engine for execution. In rare cases we will ask Gemini to actually create the final SceneMax3D code. It's possible but the results might need manual corrections to make it fit to the game.

Export to Android - the generated game should be able to run on Android devices so all kind of design aspects should be taken in consideration while generating the game's code.

So this is the high level design of the SceneMax3D-Gemini AI integration system. I believe that this design will allow people with no coding skills at all to create some fun games.

What is developed so far

Plug-ins system for SceneMax3D - the run-time engine as well as the IDE can now load pre-installed plug-ins

SceneMax3D language modifications - the programming language now supports a new "plugins" command which can start and stop any installed plugin. for example, we can open our game to accept commands from the outside world by writing the command: plugins.ws.start 8080 This means that an internal web server will start, listening on port 8080, accepting commands from the outside world via the /run REST API

In-process web server plugin for the run-time engine - allows starting an in-process web server while the game is running. For the web server support I chose using the NanoHttpd open source project. It's very simple, small and runs perfectly on any platform including mobile devices. I have very good experience using it in my Android off-road navigation application, Adv Rally where it is used for rendering an offline maps.

I tested this configuration and it works perfectly. I was able to run a scene, start the in-proc web server using the "plugins.ws.start" command and send commands to the scene from the outside world using postman client. The engine accepted the commands and executed them affecting the game's scene in real-time.

What's next

IDE plugin for converting configuration (which will be generated in the future by Gemini) to SceneMax3D code

IDE plugin for sending prompts to Gemini and accepting the results as a JSON configuration

Speech to text component - can be either a Java Swing form (implemented as an IDE plugin), an Android application or a Web application. IDE form and Web application can run locally and send their commands to the engine's in-proc server. Android app or a remote website will need an additional mechanism to push their commands to the engine. A good candidate can be socket.io client or a public SQS queue.