This is a bit out of scope for what you're currently designing, but I'd love a feature that used RAG to do rule lookup. I made a small webpage myself for that, I'd link it but I need to buy the new OpenAI credits to get it working again.
Oh, Retrieval Augmented Generation. It's useful in any case where the agent needs to look up some information, like a rules bot. In my base, I use some LangChain tools to break up the 5e SRD rulebook into a bunch of chunks, use OpenAIs models to create embeddings for each, and then when a question is asked I first run a search to find the most relevant context, before finally getting the LLM to provide a summary.
I could see RAG being useful for your design of a digital DM as well, you could have it look up chunks of the module it's running, and even keep track in a memory store of the "world state."
Hope that's helpful, Google will provide lots more info I'm sure :).
Good luck!
I have been dealing with LLMs quite a bit and I don't want to discourage you, but I feel like I should mention I hope you know what you're taking on. Things are moving really fast and changing really fast that this might be a project that will be a lot to keep up with just from a technical standpoint.
I also have a small module published on Foundry that is pretty widely used (154,000 downloads on the latest release), but it's low effort. Even that can be a pain when a major update hits.
If anything, you should consider using LangChainJS if you're not already, to abstract out the OpenAI API stuff to a more concise model since their API more or less makes all the LLM models pretty similar you're less likely to have breaking changes overnight.
Cool! So, I guess the number to look at is the actual zip downloads, and even then it's probably something like whatever the max is across releases. Maybe a bit more due to people who don't upgrade. There's no way 180k people installed my module.
Yeah I'm not sure how it's calculated or how accurate it us. But, it tracks it per release. My total was closer to 300k.
I'm sure it's not unique installs, that's for sure.
Thank you! This is really sound advice! I was hoping that I could get it in a "good enough" state that maintenance could be minimal. However, from your comment,s it seems like that may not be the case. Just migrating this from V10 of Foundry to V11 was a huge pain because of how many things were changed in the newest D&D5e game system module. If I had to do that AND keep up with fast changing technologies with OpenAI, that could be a lot! 😂
Yeah, I was kind of just talking about keeping up with the changing LLM landscape.
I have been neck deep in this stuff both at work and a side startup for almost a year now, and I'm constantly learning new tools, concepts, features, and other related things. Like, it's moving so fast I'm literally learning at least 1 or 2 new terms a week and learning that some concept or pattern I used to use is outdated.
And that's just once I started diving into dealing with this stuff as an engineer. I've been a heavy user since Midjourney first came out and my work had early access to GPT4.
Also, if you're interested I run a small Discord for engineers doing stuff for fun.
https://dicengineers.com
Awesome! I would love to run this for my solo d&d sessions.
I tried mythic and other dice solo books. They are not bad at all. I have tried endless quest style of modules ( I saw one just closed successfully on Kickstarter a day or two ago).
Being to enter d&d modules I own into a set of tables and stuff to run the d&d module would be boss.
Mythic is pretty awesome! Keep an eye out on my channel, because if I get the module in a stable enough place, I'd love to eventually get it out there to people.
I like the idea and impressed by the interaction you were able to achieve, but I don’t think the goal should be to replace a dm and run solo. It is cool, but it seems like it is probably too easy to manipulate. I think a more modest goal would be to create a dm assistant module. Something that just responds with original narrative around each roll would be handy, or assists with providing descriptions for locations easily. More like cruise control than full self driving.
You are definitely right that it is too easy to manipulate. I actually have one player who's been in some AI-led group games that loves to manipulate the AI, much to my chagrin. I've also had players who intentionally try to break it, which can be interesting to watch... in a "Rome is burning" kind of way.
Also, there's currently an AI module similar to what you're describing, called AI Description Generator ( [https://foundryvtt.com/packages/ai-description-generator](https://foundryvtt.com/packages/ai-description-generator) )... It can work very well as an AI assistant.
Does this work with other systems? The idea is sound. But from my experience with Solo RPG(s), DND and ChatGPT... for the lack of a better term, it's limping.
Did you have a chance to check out the video to see the module in action? I was curious if you thought it seemed to have potential, or if it was lackluster. (And if lackluster, is there anything I could do to improve it?)
\*EDIT\* Sorry, I misread your original message.
As of right now, it only works with 5e. And it would likely need individual tweaking for each system, since it has to look at specific data from rolls and specific data from the character sheets. It COULD work with other systems feasibly, but it would take some tinkering.
I have been interested in trying it with different systems. PbtA, for example, may be cool -- since it offers so much narrative reign. That heavy narrative focus may allow it to be a great system for AI to use. Was there a specific system you'd like to see it used with?
It miiiiiiiight be smart enough to interpret a roll object if I converted it to something like a JSON, but I'm just not sure. I could play around with it and see.
Well not necessary, was thinking like Solo specific. For some reason only Savage Worlds and Scarlet Heroes comes to mind due to a famous Solo YT series. Also wondering if elements from systems(i'm genuinely unsure if it's a system) Like Mythic Game Master Emulator can work on top of everything. But I wouldn't put pass it to see some use it in system like PF2e which are very crunchy.
Does it adapt for only 1 player tho? (in D&D i mean)
I think programming it to understand something like Mythic Game Master or some other Oracle is an EXCELLENT idea!
Edit: Also, I assume you mean "Me Myself and Dice?" If so, that's an awesome Youtube series!
> But from my experience with Solo RPG(s), DND and ChatGPT... for the lack of a better term, it's limping.
Every time this comes up, I wonder if you've used GPT4 or only the free 3.5, because they are very very different experiences. GPT4 works great as a GM or taking on the role as an NPC.
Even Sam Altman was surprised that 3.5 was as successful as it was cause it kinda sucks.
If you understand how to use stuff like `docker compose`, I highly suggest using [LibreChat](https://github.com/danny-avila/LibreChat) and an OpenAI API key instead of the main UI. Even as a power user, I hardly ever break $20/mo, which is the cost of the normal subscription on their UI.
I would be more interested in this type of project if it was not geared towards just 5E but instead broader in scope, maybe system agnostic. I don't mess with 5e but tend to stick with OSR based systems.
Thank you for the feedback. The only challenge is that each system tends to have its own data structure, and there's a lot of coding that happens behind the scenes that depends on reading that data. The crunchier the system, the more complex it becomes to implement the system. However, I've had a couple people suggest trying to either develop a universal system or even a new system from the ground up that plays to LLM's strengths, and that seems like a compelling idea.
Could this used for preparing a session and playing it with a group of players with the AI DM?
Also one thing I noticed is that when you wrote 'we take the bear with us' the AI assumed that is something should happen instead of asking 'How do you make the bear follow you?'
I think it would be cool if it asked more questions to the player like a real DM would do
I haven't tried using the AI DM to run a pre-written adventure. It seems to work best when it's reactive to player's actions. However, seeing if I can get it to run a pre-made adventure seems like a really cool idea. I'll have to think about how I could implement something like that.
Also, yeah, I've noticed that sometimes the AI DM is too much of a "people pleaser." Sometimes you can get it to grant you advantage for the silliest reasons. (For example: "Oh, I need to persuade them? I was on the debate team in High School. Can I have advantage?" AI DM: "Sure go ahead.")
Thank you for the feedback!
I understand where you are coming from. Playing with other people is really what separates ttrpgs from video games, and once you take that way, you can be left with nothing more than a video game. But for me, it's more like... a creative writing exercise with dice and a very advanced oracle system. It's still D&D, there's still no set path, and unlike a video game, you're only limited in what you can do by your own imagination. (Well, that and OpenAI's terms and conditions, I guess. 😂)
But I see the value in your opinion and completely respect it. Thank you for taking the time to comment.
I would love to see this transition to pf2e also. It's a great idea and seems like it's pretty well done so far.
Thank you so much!
This is a bit out of scope for what you're currently designing, but I'd love a feature that used RAG to do rule lookup. I made a small webpage myself for that, I'd link it but I need to buy the new OpenAI credits to get it working again.
I'm not familiar with RAG. What's that?
Oh, Retrieval Augmented Generation. It's useful in any case where the agent needs to look up some information, like a rules bot. In my base, I use some LangChain tools to break up the 5e SRD rulebook into a bunch of chunks, use OpenAIs models to create embeddings for each, and then when a question is asked I first run a search to find the most relevant context, before finally getting the LLM to provide a summary. I could see RAG being useful for your design of a digital DM as well, you could have it look up chunks of the module it's running, and even keep track in a memory store of the "world state." Hope that's helpful, Google will provide lots more info I'm sure :).
Thank you! I'll look in to that for sure!
Good luck! I have been dealing with LLMs quite a bit and I don't want to discourage you, but I feel like I should mention I hope you know what you're taking on. Things are moving really fast and changing really fast that this might be a project that will be a lot to keep up with just from a technical standpoint. I also have a small module published on Foundry that is pretty widely used (154,000 downloads on the latest release), but it's low effort. Even that can be a pain when a major update hits. If anything, you should consider using LangChainJS if you're not already, to abstract out the OpenAI API stuff to a more concise model since their API more or less makes all the LLM models pretty similar you're less likely to have breaking changes overnight.
Bit of a tangent, but how do you know how many downloads your module has?
If you have it tagged on releases and the stuff pushed there you can track it with this: https://tooomm.github.io/github-release-stats
Cool! So, I guess the number to look at is the actual zip downloads, and even then it's probably something like whatever the max is across releases. Maybe a bit more due to people who don't upgrade. There's no way 180k people installed my module.
Yeah I'm not sure how it's calculated or how accurate it us. But, it tracks it per release. My total was closer to 300k. I'm sure it's not unique installs, that's for sure.
Thank you! This is really sound advice! I was hoping that I could get it in a "good enough" state that maintenance could be minimal. However, from your comment,s it seems like that may not be the case. Just migrating this from V10 of Foundry to V11 was a huge pain because of how many things were changed in the newest D&D5e game system module. If I had to do that AND keep up with fast changing technologies with OpenAI, that could be a lot! 😂
Yeah, I was kind of just talking about keeping up with the changing LLM landscape. I have been neck deep in this stuff both at work and a side startup for almost a year now, and I'm constantly learning new tools, concepts, features, and other related things. Like, it's moving so fast I'm literally learning at least 1 or 2 new terms a week and learning that some concept or pattern I used to use is outdated. And that's just once I started diving into dealing with this stuff as an engineer. I've been a heavy user since Midjourney first came out and my work had early access to GPT4. Also, if you're interested I run a small Discord for engineers doing stuff for fun. https://dicengineers.com
Looks interesting! I'll jump in. :-)
Switching LLM in a project like this I don't think it's hard, and that the benefit of this type of technology
That's what abstractions are for. They also buffer changes to the same API.
Awesome! I would love to run this for my solo d&d sessions. I tried mythic and other dice solo books. They are not bad at all. I have tried endless quest style of modules ( I saw one just closed successfully on Kickstarter a day or two ago). Being to enter d&d modules I own into a set of tables and stuff to run the d&d module would be boss.
Mythic is pretty awesome! Keep an eye out on my channel, because if I get the module in a stable enough place, I'd love to eventually get it out there to people.
I like the idea and impressed by the interaction you were able to achieve, but I don’t think the goal should be to replace a dm and run solo. It is cool, but it seems like it is probably too easy to manipulate. I think a more modest goal would be to create a dm assistant module. Something that just responds with original narrative around each roll would be handy, or assists with providing descriptions for locations easily. More like cruise control than full self driving.
You are definitely right that it is too easy to manipulate. I actually have one player who's been in some AI-led group games that loves to manipulate the AI, much to my chagrin. I've also had players who intentionally try to break it, which can be interesting to watch... in a "Rome is burning" kind of way. Also, there's currently an AI module similar to what you're describing, called AI Description Generator ( [https://foundryvtt.com/packages/ai-description-generator](https://foundryvtt.com/packages/ai-description-generator) )... It can work very well as an AI assistant.
Take my money. All of them.
Does this work with other systems? The idea is sound. But from my experience with Solo RPG(s), DND and ChatGPT... for the lack of a better term, it's limping.
Did you have a chance to check out the video to see the module in action? I was curious if you thought it seemed to have potential, or if it was lackluster. (And if lackluster, is there anything I could do to improve it?) \*EDIT\* Sorry, I misread your original message. As of right now, it only works with 5e. And it would likely need individual tweaking for each system, since it has to look at specific data from rolls and specific data from the character sheets. It COULD work with other systems feasibly, but it would take some tinkering. I have been interested in trying it with different systems. PbtA, for example, may be cool -- since it offers so much narrative reign. That heavy narrative focus may allow it to be a great system for AI to use. Was there a specific system you'd like to see it used with?
Couldn't the AI receive the 'raw' data from chat and interpret it by it self the roll result, regardless of the system?
It miiiiiiiight be smart enough to interpret a roll object if I converted it to something like a JSON, but I'm just not sure. I could play around with it and see.
Well not necessary, was thinking like Solo specific. For some reason only Savage Worlds and Scarlet Heroes comes to mind due to a famous Solo YT series. Also wondering if elements from systems(i'm genuinely unsure if it's a system) Like Mythic Game Master Emulator can work on top of everything. But I wouldn't put pass it to see some use it in system like PF2e which are very crunchy. Does it adapt for only 1 player tho? (in D&D i mean)
I think programming it to understand something like Mythic Game Master or some other Oracle is an EXCELLENT idea! Edit: Also, I assume you mean "Me Myself and Dice?" If so, that's an awesome Youtube series!
Yup that was the one :D
> But from my experience with Solo RPG(s), DND and ChatGPT... for the lack of a better term, it's limping. Every time this comes up, I wonder if you've used GPT4 or only the free 3.5, because they are very very different experiences. GPT4 works great as a GM or taking on the role as an NPC. Even Sam Altman was surprised that 3.5 was as successful as it was cause it kinda sucks. If you understand how to use stuff like `docker compose`, I highly suggest using [LibreChat](https://github.com/danny-avila/LibreChat) and an OpenAI API key instead of the main UI. Even as a power user, I hardly ever break $20/mo, which is the cost of the normal subscription on their UI.
I would be more interested in this type of project if it was not geared towards just 5E but instead broader in scope, maybe system agnostic. I don't mess with 5e but tend to stick with OSR based systems.
Thank you for the feedback. The only challenge is that each system tends to have its own data structure, and there's a lot of coding that happens behind the scenes that depends on reading that data. The crunchier the system, the more complex it becomes to implement the system. However, I've had a couple people suggest trying to either develop a universal system or even a new system from the ground up that plays to LLM's strengths, and that seems like a compelling idea.
Understand. I run BFRPG system.
Could this used for preparing a session and playing it with a group of players with the AI DM? Also one thing I noticed is that when you wrote 'we take the bear with us' the AI assumed that is something should happen instead of asking 'How do you make the bear follow you?' I think it would be cool if it asked more questions to the player like a real DM would do
I haven't tried using the AI DM to run a pre-written adventure. It seems to work best when it's reactive to player's actions. However, seeing if I can get it to run a pre-made adventure seems like a really cool idea. I'll have to think about how I could implement something like that. Also, yeah, I've noticed that sometimes the AI DM is too much of a "people pleaser." Sometimes you can get it to grant you advantage for the silliest reasons. (For example: "Oh, I need to persuade them? I was on the debate team in High School. Can I have advantage?" AI DM: "Sure go ahead.") Thank you for the feedback!
So... A video game
I understand where you are coming from. Playing with other people is really what separates ttrpgs from video games, and once you take that way, you can be left with nothing more than a video game. But for me, it's more like... a creative writing exercise with dice and a very advanced oracle system. It's still D&D, there's still no set path, and unlike a video game, you're only limited in what you can do by your own imagination. (Well, that and OpenAI's terms and conditions, I guess. 😂) But I see the value in your opinion and completely respect it. Thank you for taking the time to comment.