Conversational Interactive Agents Project @conversational-agents-2018 - Tumblr Blog

JSON Pretty Text

As our target audience is researchers, an important part of this project is being able to quickly edit agents from the JSON files containing the tree data. For this reason, the output of saving the tree is now formatted as pretty text for readability. I’ve shown this in the first two images of this post, with the first picture being regular format, and the second picture being pretty text format.

Pretty text format does not format correctly in the app often used by non-tech people (notepad) but this issue is low priority, so I will only be attempting to write a fix for this when all other components of my section of the prototype are complete.

#process

Proposal Update Week 7

This week I added some new functionality to my prototype, as well as creating reference documentation for it.

Changes

There are now references to ‘connection nodes’ in the classes DialogueNode and DialogueTree, as can be seen in the UML above. Connection nodes are nodes that have a child node, but exist as a connection through the story and the player can not actually interact with the agent on this node, similar to clicking through text in an RPG. This was discussed in the meeting held this week.

Activation functionality is now somewhat implemented, as the DialogueTree tracks the ID of the currently active node.

Future Plans

When we are further into the project, I’d like to implement functionality for variable tracking throughout the dialogue tree and tracking of agents/their animations and reactions.

#processs #proposal

Reference Documentation

As there are two programmers on this project, and to have a reference for the back-end of the project, I will be writing up a reference guide for my code. Before beginning this, I researched how others had implemented this kind of documentation and the most easily understandable implementations.

The three main reference documentation sources I am familiar with are the Unity manual, the .NET documentation, and the Python documentation.

Of the three, I have found that the Unity manual is the easiest to understand, as it lays everything out in simple terms under big headings. Python and .NET references are explained in multiple paragraphs (as seen in pictures 1 and 3), while the Unity manual has basic sentences, that are explained succinctly in their own pages.

In my experience, the .NET guide is easier to understand than the Python guide, as although it explains everything in detail, which results in an overwhelming amount of text, it is interspersed with a lot of examples. In many cases, it has a lot more examples than the Unity Manual.

Take Away and Thoughts

From this research, I plan to:

Use a lot of subsections to break up the references, similar to the Unity manual

Use a lot of examples, similar to the .NET documentation

#research

Meeting Week 7

In our week 7 meeting we went through the work each member had done. After this meeting, I plan on modifying/implementing nodes so that they can tell whether they are termination nodes and they can have null input and simply have a relation that leads to the next node.

I will also be researching gitignore files for Unity and writing up an API dictionary for my code for use by the other programmer.

#process

Proposal Test Implementation

In week 6, I implemented my planned proposal. It went pretty smoothly, however, after testing of the implemented structures and functions (seen below), it became apparent that the UML required a few changes and extra classes and functions.

Allowing for Serialization

While testing my implementation of my proposal, I noticed that almost no fields were being serialized. I initially did some additional research into what kinds of fields could be serialized. From this, I discovered that only public variables could be serialized, meaning I had to change all variables to become public. This, however, still left some variables out, as seen in the below output of a test:

I then realised that an additional mistake had been to mark the classes as [SerializableField], rather than:

After applying this to all classes,

Change to Serializable Dictionary

As seen in the above image, the changes I made to allow for serialization worked on the most part - but not for fields of type SerializableDictionary<K, V>. After some testing, it became apparent that the reason for this was the variable data types, which lead me to create the two new classes, PlayerChoiceDictionary, which holds the text and childIDs of the choices for each node, and DialogueNodeDictionary, which holds nodes and their IDs. These inherited from SerializableDictionary, and after their implementation, a save of the tree resulted in the below output, which accurately records all tree information.

JSONWrapper

Finally, I also added a JSONWrapper class to the UML, as I felt that the saving and loading functionality of the tree should be contained in a seperate class to the tree itself. Additionally, allowing it to be a static class meant its functionality could be accessed more easily.

Take Away

From this test, I’ve found that the goal of implementing the entire back-end for the prototype should be easily achieved by week 13. I feel the changes I made were very useful for the final product, and I hope to soon build on this base after a discussion at the team meeting on Monday.

#process #proposal

Proposal: Error Checking Update

As mentioned in my previous workshop post, I have updated my current proposal to include error checking functions: HasChild() and IsEmpty() in DialogueNode which check for the indicated attribute, and HasInitial() in DialogueTree, which checks if it has an initial node.

The DialogueTree class also now has a new attribute too track the relations between nodes.

#process #proposal

Workshop/Meeting Week 6

Meeting

This week our meeting with our mentor was called off, and although it later got called back on, myself and another group member had already arranged a different meeting with our capstone group. This meant that our meeting to discuss the project was somewhat held during the workshop, as can be seen in my notes which are a mixed set of meeting and workshop notes.

We mostly discussed the GUI, as Geordie had created the designs for it. On the first page, I noted what functions I had to add to my current UML that I had not thought of but had become apparent when going through the intended design. These are mostly error-checking functions, and I plan to edit my UML to contain them before beginning my test implementation of it.

Workshop

We went through a few things in the workshop but for me the most important was going through the presentation CRA. We discussed what we would need to include in it as well as how it would be laid out. I only thought of one scenario before the class ended, but plan on researching more other similar videos in week 7 to brainstorm more video ideas.

#process

Proposal #2: Initial Back-End Proposal

Summary

For my initial proposal I have designed the base of my back-end implementation. It does not currently allow for agent response beyond text (animations, sounds), which we have planned for the project, nor does it allow for numerical tracking. However, it currently takes into account the majority of back-end interactions, as can be seen in the first image. I plan on implementing this framework and testing it through JSON files before attempting to iterate. After I have Implemented this base I will be writing up API documentation so that the UI programmer will be able to use it more easily.

Initially, as seen in the fourth image, I split the node class into two classes. I did this, as seen in my planning notes, as I initially planned for a single node class, then attempted to consider how this class would be interacted with by the user. However, reviewing the UML, I felt that this was unnecessary and collapsed them into a single class in the final base proposal.

I also changed the initialisation of the Dialogue tree between the initial and final base proposals so that it could be initialized without a base node. I felt this was necessary for the user to be able to switch which node is the initial one and to create and name a tree before having any nodes.

Inspiration

I mostly based this proposal around the tasks required by the brief with some inspiration from the DialogueSmith extension, as mentioned in the linked post.

Scope and Feasibility

As I have attempted to make this proposal as basic as possible and left out some extra features, I feel that the scope is small and that I will be able to complete the implementation of this base in a short amount of time in order to iterate on it.

#proposal #process

DialogueSmith

I found a unity extension on github called DialogueSmith, which does a similar task to my section of the project - creates and saves a dialogue tree that is editable by a user - in this case the person using unity.

This project was not very well documented, there are very few comments throughout the code and there are also many scripts with very similar names. The UI aspect is also present throughout the code, so it was not exactly the same as our project, however, it was very useful as a case study and understanding how I should best structure my code.

Take Away and Thoughts

From DialogueSmith I realised I should:

Separate dialogue tree nodes into two classes - back-end Node data and the actual node script, which would go on the node window and update it’s own back-end node data

Create my own serializable version of the dictionary data type to use when storing relations between nodes

#research

Setting Up Git

The majority of the members of the development team, including myself, don’t have a lot of experience with using git to share the project, so I had to research the use of it a little bit. I watched a few tutorial videos, as well as read through a text tutorial using the command line. After I felt confident in my understanding of the basic terms and process of using git repositories, I set up the base repo for this project, as can be seen in the second image above.

#process

Data Serialization Research

For the first part of my research I wanted to understand data serialization, as I have only used txt files for storing data through code before, and I will be using it to store node data when implementing the back end of the project.

What is Data Serialization?

I first wanted to understand data serialization and how it worked. I watched a tutorial video about serializing through unity, as well as reading a tutorial article about the importance of data serialization and what it is.

From these sources I learnt that data is serialized by converting objects to bytes, storing that data in documents, and then reading it back into the application later. This is useful for allowing data to be read by other platforms (e.g. allowing a web server running on java to understand a C# object by serializing it then allowing the server to deserialize it) and saving a large amount of data, such as a save file.

What Kind of Data Serialization Should I Use?

The kinds of data serialization I read about were XML, JSON, and YAML. JSON is shorter format than XML format, easier to parse by humans, and allows for arrays. XML also, according to the video tutorial previously mentioned, requires a lot more code and processing to read and write to than JSON in Unity. Finally, there is a lot of easy-to-find support for using JSON online. Therefore, the decision between XML and JSON was clear.

Next, I chose between JSON and YAML. YAML seemed to have better reviews, as people liked the ability to comment in YAML files (unavailable in JSON), as well as tagging and anchoring certain data. One article recommended using JSON for communications between clients and servers, while using YAML for human-readable documentation. However, while researching Unity implementations of YAML, I found that while Unity has built-in support for JSON, YAML would require extra installations and code for only a small difference in output - especially since I do not really need to comment or reuse variables in this project.

Learning How To Serialize Data

After choosing a file type to use, I learnt how to read and write to, as well as create, JSON files using a video tutorial and some documentation. The results of this research can be seen in the pictures at the beginning of this post. The code seen in these pictures contains the previously mentioned functionality, and I now feel confident in using data serialization for this project in the future.

#research #process

Team Meeting Week 5

This week we further discussed roles moving forward and the actual beginnings of our project. The group was split into two (designers and animators with Nicole, programmers with Gavin) and we discussed our roles moving forward and how we would start work on the project. Gavin introduced us to using git (both myself and Christo have only really used Unity Collaborate before) as well as using JSON/YAML files for data storage (I’ve only really used txt files before).

After this meeting, I feel I have a clear set of tasks to complete moving forward:

Set up the git repository for the project

Research and understand using JSON/YAML files

Plan the data structures for the back end of the project

Create API documentation so that Christo (front-end programmer) can clearly understand what every function does

#process

Team Meeting Week 4

On Monday, my team had our first meeting together in a room - our first in week 3 was held over call. In the meeting we formally split up the tasks of the project, as well as discussed the ‘innovation’ aspect of the project, as our mentor did not seem fully aware of this aspect of the project.

After the meeting, everyone besides the programmers came away with their own separate and clear goals. The two programmers (myself and Christo) did not have our goals clearly split as the two overarching jobs involved separate into ‘back-end’ and ‘front-end’. For now, for the most part, I am ‘back-end’, but our jobs may overlap. Initially this was worrying as our class mark supposedly relies on individual innovation, which we both felt it may be difficult to implement in terms of front-end programming. However, after discussion with tutors, we have been informed that significant individual innovation is not as important in our project, as long as out contribution is clear.

Because of this, in the future of this project I will likely focus more on prototyping and iterating on the current proposal than creating new proposals. I will continue to research similar systems, but going forward there will be a heavier focus on the back-end of these systems.

#process

Proposal #1: Procedural Generation of Agent Responses Based on Tracked Variables

Summary

For this proposal, I intend to allow the user to either create an initial behaviour tree base or use a predefined one, based around some variables, then give them drag and drop boxes to attach these variables to dialogue segments. Using these two inputs the program will automatically generate agent reactions to dialogue choices, without the user having to manually input the reaction to each dialogue choice, which they can also edit through the dialogue tree. Users would be able to name and track variables throughout the dialogue trees, with a list of variables that will highlight all instance of a variable when one instance is selected, as shown below.

Inspiration

This idea was inspired by my research into both the TW3 cinematic sequence system and the Twine engine.

The use of scene generation in the TW3 system inspired me, but since this project is to be developed over 10 weeks it is unrealistic to aim to create a system as complex as the one in the TW3 system. However, the generation tool provided simplicity and usability to a wider set of users who may not be as skilled at manipulating technical tools. Taking inspiration from the look at system in the TW3 system, in which code interpolates between animator-input poses to generate a system that will always work, I considered the possibility of doing the same thing with agent behaviour. Rather than creating code that would account for all possibilities, the agent behaviour could be a collaboration between code and user, in which the user would predefine variables and agent reactions to variables, then modified variables at certain points in the dialogue tree to invoke different agent reactions.

The Twine engine inspired my thoughts on extending the use of variables in this project. The initial brief did want the agent to account for numerical value tracking - but Twine allows for variable tracking of many different types with minimal coding. Due to this, I think that many variable types could by implemented through a drag and drop interface without much confusion.

Scope and Feasibility

Variable tracking and implementing drag and drop values is simple, but may be difficult to integrate depending on the split of programmer tasks in this project

Generating behaviour is a difficult task in itself, so it may be out of scope to create a system that generates behaviour with any kind of complexity

As long as it is well designed and documented, a vary simple behaviour generation system may be sufficient

Behaviour of an agent will depend on animations

If this is the only modification to the current objectives, I feel that it is feasible, but adding other elements will make this section out of scope

Impact on Users and Stakeholders

Users will have more freedom to create the behaviour of an agent without needing to input its reaction to every dialogue segment

#proposal

Proposals

For my section of the project I have a very clearly defined outcome, as detailed in the about page of this blog, so when I make proposals I will be detailing changes/additions I intend to make to the current expected outcome. These posts will be numbered and detailed in the following template to ensure understanding:

Proposal #Number: Proposal Name

Summary

Inspiration

Scope and Feasibility

Impact on Users and Stakeholders

#process

Kodu Engine

Of the three systems researched, I think the Kodu engine is the least relevant. It has an interesting UI designed to be simple and usable by anyone. However, it was specifically designed to teach programming to novices and some reviews doubt its success in this intention. In terms of interactive agents, there is no way to implement dialogue trees outside of simple text triggered by actions. Due to this and the fact that I have already discussed the UI implemented in the Twine engine, this post will be brief.

Similarities

Drag and drop blocks to guide agent

Simple numeric tracking (the score counters)

Simplistic UI that theoretically allows non-tech minded people to pick it up easily

In my own experience with it, this is often unsuccessful as it can be a difficult program to wrangle even for programmers

Animations and models are premade

They have simple customisable options - different colours

Useful for non-tech people to not have to go through the process of animating/modeling

Differences

Programming/Game creating focused

Little customisability - a simple UI that can be restrictive

Can’t create an interaction based output as conversation options are limited

Outcome is teaching user to program/ outcome of this project is to create and agent for use by researchers

Models are not modular and have separate move/ animation sets

Can be annoying for users who want to choose/create a model based on aesthetic alone

Take Away and Thoughts

I did not gain a lot of new insight from this research, but it did help me to understand:

Focusing too heavily on a simple UI and not leaving room for complexity can leave the user feeling frustrated

I spoke about the benefit of integrating simplicity and complexity in my earlier Twine post, however the Kodu engine has changed my view to think complexity is more necessary than optional - which i speak about in my first proposal

Variable tracking can be made simple, but should have an element of complexity to be understandable

Kodu splits up numeric tracking based on colour, which can be unintuitive and confusing to keep track of

I would like to do something similar to Kodu’s value tracking while allowing the user to name values and have more data types than just integers

#research

Twine 2.0 Engine

The Twine engine is used to create text-based choice-oriented games. While the outcome is not the same, the way dialogue trees are handled is an interesting topic that could provide insight into the flow of the conversational agents project. As in my previous post, I will be discussing the similarities and differences between this project and the Twine engine, as well as my take away and thoughts.

Similarities

An easy-to-use drag-and-drop UI for the creation of an option tree

Easily accessible for kids and non-tech people

Customised text

The ability to use hidden systems to keep track of player choices

The ability to customise the player’s view/knowledge of the scene

Customise reactions of ‘agents’ (in this context they are mostly just written but Twine has the ability to insert images too)

Allows the player to guide the decision stream through the use of selection buttons

Differences

There are no predefined blocks - story is created completely from scratch

There is no real ‘agent’ that is being interacted with with animated responses

There is no design of a behaviour tree, as it is more of a story engine than an engine to create an interactive agent

HTML can be used to customise text/scene look and feel/other effects

Things can happen in the story without the player interacting with the scene - the engine is extremely customisable

Take Away and Thoughts

This system has given me some understanding of how the dialogue trees should work in this project. I now understand:

Variable tracking can be extremely useful in designing the behaviour and results of interactions

It could be useful to include options for more than simply numeric value tracking - different options could be provided

I would hope to implement this in a more easily understandable way than Twine, as it requires a basic knowledge of coding to understand, as seen in the second image

Variables could be drag and drop boxes that are attached to dialogue segments, with drop down fields for what happens to the variable at each segment

Could have clutter issues when tracking many variables

A simple UI with complex features can be implemented

In my previous research post, I discussed the need to compare and weigh the importance of both, however twine does have both

A good game can be made with the default Twine settings, with only script text and very basic commands used

However, users can also apply a lot of customisation through the use of HTML

This kind of system could be designed for use in this project, as discussed in the TW3 research post

It could be beneficial to the user to implement a system by which the agent would react to certain events without the end user choosing a dialogue option

Similar to HTML use in Twine to make certain things happen without the player specifically choosing an option

Would use the behaviour tree section of the conversational agent

#research

Trending Blogs

Recently Viewed Blogs

Conversational Interactive Agents Project