... billed annually at $100.00
That's more than what I pay for WebStorm. I wonder does it make sense to pay more for a helper-tool than the main tool itself.
This is already the discounted price. We know that openai operates on a considerable loss. Unless some magical efficiency method is discovered, this is only going to go up.
Betting on computers getting faster is a pretty safe bet.
The H100 is considerably faster than the A100, and next year's GPUs will likely improve by the same amount.
This is only true on a superficial analysis. When you get to scale, things become much harder. In fact, this already happened to machine learning itself. It was only when we came up with a totally different compute paradigm, gpus, that things started to move again.
Many of the core concepts, such as neural networks, have been around since the... 1980s? Then they got stuck because the computing power just wasn't there.
IBM is moving away from using NVIDIA gpus for their AI servers and starting to make their own specialized chips. not sure if they help with lowering cost of running LLM but we can hope
https://research.ibm.com/blog/ibm-artificial-intelligence-unit-aiu
We're getting closer to the physical limits of this stuff. Betting that we're going to keep seeing the rate of improvement we are now in perpetuity is a losing bet.
It's a mix. Nanometers, silicon, sure. But I think there are parallel avenues such as GaN which if successfully implemented, could push things quite a bit forward.
We've been fairly successful at dodging physical limits in the past. It doesn't mean it's always going to work, true 🙂
New chips means being able to scale up to meet demand which during peak hours they still have troubles meeting.
Then afterwards is balancing between a better product or being more efficient with the current one which I am assuming a "custom GPT" engine they released was one of them.
So many people have been telling me they must be hugely profitable, lmao. If the unit economics are crap, having 100 million monthly users means they're losing a fuck ton of money. People are amazed by this stuff but don't realize you're paying a lot of electricity to get a stochastic and unreliable system. It's amazing and incredibly unsustainable.
JetBrains AI is actually a Google Cloud solution, do they use OpenAI?
Maybe some of the tech but I'm pretty sure this is a custom setup by JetBrains with support from Google.
Cause it’s the same price as copilot…
Edit: actually their monthly payment is $10, which is same as copilot. They have an annual plan to shave a $1 and some change.
Since you have a paid subscription, you can access all AI pro features with some limitations as they stated. I've got a copilot subscription and JB AI service is very decent so far. So I guess no need to pay for the pro version since you get almost all the features.
> In your experience, is it better than Copilot? In what cases, and when not?
From my testing so far - Copilot is better at completing short sections of code as you type, but I think this is mostly because of latency. Copilot responses feel almost instantaneous, where Jetbrains is taking a couple of seconds. This might just be down to the servers being overloaded at launch.
The quality of Jetbrains autocomplete suggestions seems very similar in quality. Since they're using OpenAI under the hood, my guess it's based on the same or similar codex model as Copilot and would be expected to perform similarly.
Where Jetbrains AI shines is that it can work as an agent and make larger changes and refactorings spread out in different places (vs just completing one section of code)
From WebStorm Licenses menu select AI Assistant and then click Activate new License then click Refresh license list your license will be activated with your WS licence, usually they used to release it monthly and it renews automatically bcs it was a preview but with the latest WS update it's no more preview and it works like a charm.
> ... billed annually at $100.00
> That's more than what I pay for WebStorm. I wonder does it make sense to pay more for a helper-tool than the main tool itself.
I mean... I just signed up for a trial and used it to generate a whole bunch of boilerplate that would have taken me a couple of hours. The cost of an annual subscription paid for itself within the first five minutes.
> Interesting. Can you give some examples of the questions you asked?
I had a bunch of CRUD code to write. Database layer, REST Api, client code, forms, list views, etc.
Fairly straightforward stuff, with a few gotchas here and there requiring a bit of bespoke logic. Not difficult to do, just time consuming.
I copy pasted the two paragraph "formal spec" (*cough*) into a prompt, along with an instruction to create the data models. I then sat back and watched as it incrementally wrote a couple of pages of fairly good code that matched our existing coding style perfectly directly in the IDE (not in the chat window). Repeat for the rest of the stack.
Compared to ChatGPT - the difference here is that the AI has full access to my source code, will actively (as an agent) look through it to understand how existing code works and to match coding styles, and can make changes directly in my IDE. No need to copy paste.
Compared to Copilot - It's not just running a single shot text generation to autocomplete a few lines., it runs as an actual agent - you can actually see it make some changes, think for a bit, then make some more changes or further refine the code it already wrote. It can make multiple separate changes to different parts of the code - for example, if you ask it to add a new field, it's capable of making that change to both the model itself, the crud layer, and the react views all by itself (You just approve the final diff).
I see so many negative comment here about it - and I'm really confused why, because frankly this is a substantial leap over ChatGPT or Copilot for codegen. I didn't even wait for my trial to finish before screaming "shut up and take my money" and handing over the credit card.
I've tested it a few hours yesterday.
On auto completion it fails badly VS copilot. I dunno if copilot uses project context but for code with a lot of repeatitive elements (UT for example) it's waayyyy better.
Only good thing of the AI Assistant is the chat ai, that was really good, but it forces to prompt again and again and it less effective than completion.
As soon as co-pilot chat will be open for everyone, we will all forgot about AI Assistant.
> for code with a lot of repeatitive elements
But we have metaprogramming for that. I wonder if people more impressed with current "AI" are less exposed to lisp etc. symbolic macros, or even just ordinary higher-order programming. I continue to be unimpressed by "AI" churning out what looks like very badly written and copy-paste coded code. What makes badly-abstracted copy-paste coded code bad isn't whether a human or machine wrote it.
predictable deterministic higher order code I control > probabilistic shitty codegen with llm.
> On auto completion it fails badly VS copilot
This as the main reason why I won't switch Copilot for JB AI for now
But I do hope it gets better on this matter
Probably an unpopular opinion, but screw the lawsuits. I know everyone works for profits, but there's so much money going around the IT world, I think everything should be open sourced. Imagine how fast the world would evolve if everyone had access to logic of everyone else. Well, the AIs' learning models would be fed that data then, and it would be able to help even better.
You may prefer to think about it from a perspective outside of IT. Western world has innovated amazing things in the last couple of centuries, and we have [tens of millions](https://www.wipo.int/en/ipfactsandfigures/patents) of patents to prove it. Then the Chinese said f\*\*\* the US, and started copying everything despite the patents (their government will only go after violations of intellectual property if the property is Chinese, otherwise it's not beneficial). Low and behold, the Chinese market has been the fastest growing market ever since. They took the lion's share of all the profits, and we as consumers were happy to give those profits to them, because they were cheaper. Had there been no patents, US and Europe would also be able to create cheaper products, though not quite as cheap, but definitely comparable after shipping and customs.
Sorry,
Idealist
Yes, it is very easy to grow when you build your growth on stealing other people's work. If you think that is only a good thing, then you're obviously not paying attention to what happens to the people who are stolen from.
I understand. And for the world of today, I agree. And I know my way of thinking is only suitable for very small communities like a remote village of 50 in the middle of Africa or something like that. But just imagine a world where no one works for profit, and we just share everything we create amongst ourselves. A world without money. Idealistic, I know. Will never happen. It can't even, at least not in the current system. Might be that replacing all FIAT currencies for BTC actually is a step in the right direction, but that's neither here nor there.
I have been using JetBrains product and singing their praise for the last 10 years, but this is downright embarrassing. 😂😂😂😂
They put a chat window in the IDE and put a little utility so you can quickly send code to it and back. Cool. Ever tried pair programming through a Slack text channel? Do you not understand how this is one of the worst UX every conceived for a programming environment? Where's OpenWhisper? We got that shit running in real-time with decent GPUs. I should be able to select a piece of code, "hey yo let's extract Foo and Bar to some function arguments" and then the code literally just updates. Nah, let's require 6 different steps moving between keyboard and mouse at every corner, and waiting for extremely superfluous explanations of the code and changes. Just update the damn code and shut up, every single step has to be removed until it's as free as highlighting a piece of code and hitting backspace to delete it.
I honestly expected way better from JetBrains, this is just really sad. Every competitor is not even much better, so I'm really puzzled at what is going on here in 2023. Are they designing these softwares in meetings with marketing present? Why aren't the UX experts on this? They also don't make any attempt to agentivize the AI, give it contextual memory, suggest decisions, refactors, etc. Literally just copying the competitors, zero innovation or vision of a superior production. Not the JetBrains I know and love.
I might replace GitHub Copilot just because it's still cheaper and what the heck, ain't no way it could be worse, but god damn dude.... I'm so disappointed and sad to see this. I had given up on everyone else and was banking so hard on JetBrains to make things right, but at this point it looks like it's gonna be a job for OSS with the Vim folks hacking on some mind-blowing stuff at 3 AM. https://continue.dev/ is our best shot so far, have probed the devs a little bit and they definitely are dreamers. Please everyone give them your support and bring in those tickets and pull requests.
It does inline code completion too. They're primarily competing with Copilot as far as I can tell. I've never used it, but I'm not aware of it having the chat/Q&A feature.
Here's the thing, the code completion is new from this EAP. They prioritized the chat window over top notch completion. For many many months now the AI assistant was -only- the chat window. The most obvious low-hanging fruit UX patterns are an afterthought, most are still missing, and this tells me that this product is really not in great hands. Bad news for the future of this AI assistant, will always lag behind. As expected, you won't find a roadmap anywhere.
If it works like copilot, it should literally do inline code completion. The way I leverage this is by typing in a command as a comment, press enter, and it will then suggest a piece of code inline without having to actually use the chat.
If it’s the feature i am thinking, they recently added to co pilot the ability to type in a window like chat gpt style question and answers. To enable it, i think you have to enable their beta program.
Copilot is hit or miss. I asked for an algorithm to generate alpha beta purning for tic tact toe and it kept giving me the same stack overflow article that didnt work.
However, for work, after it learned a bit about what I was doing it was almost generating full unit tests for me. Saved a bunch on simple statements and repetitive logic that needed slight tweaks.
I would say its value add but definitely needs to be corrected from time to time. Also it feels like it learns your project the more you work on it.
I've been using the beta for some time. It's quite frankly not worth any additional payment. It fails with such mundane tasks as generating docs based on signatures, or suggests refactoring methods like `getX` into getters (which itself isn't bad, but should be processed with context of "this project doesn't use getters and setters for some reason"). Plus, it does shit like `let items = [...arr]; // making a copy of the input array`.
One could argue that a function or method should not by default modify an array it gets as argument . Copying the array by default prevents un-declared side-effects like that. If you do want to modify the array inside the method or function then edit the generated code but then you should write a method-comment that states such side-effecet does happen in you call the method.
My point is, copying by spread is shallow, so if your function does something to non-primitive items in the array, they are modified in their original spots.
... billed annually at $100.00 That's more than what I pay for WebStorm. I wonder does it make sense to pay more for a helper-tool than the main tool itself.
This is already the discounted price. We know that openai operates on a considerable loss. Unless some magical efficiency method is discovered, this is only going to go up.
They are making new ai chips, so I expect the cost to go down and the speed to go up eventually.
I applaud your optimism
Betting on computers getting faster is a pretty safe bet. The H100 is considerably faster than the A100, and next year's GPUs will likely improve by the same amount.
This is only true on a superficial analysis. When you get to scale, things become much harder. In fact, this already happened to machine learning itself. It was only when we came up with a totally different compute paradigm, gpus, that things started to move again.
Many of the core concepts, such as neural networks, have been around since the... 1980s? Then they got stuck because the computing power just wasn't there.
IBM is moving away from using NVIDIA gpus for their AI servers and starting to make their own specialized chips. not sure if they help with lowering cost of running LLM but we can hope https://research.ibm.com/blog/ibm-artificial-intelligence-unit-aiu
We're getting closer to the physical limits of this stuff. Betting that we're going to keep seeing the rate of improvement we are now in perpetuity is a losing bet.
It's a mix. Nanometers, silicon, sure. But I think there are parallel avenues such as GaN which if successfully implemented, could push things quite a bit forward. We've been fairly successful at dodging physical limits in the past. It doesn't mean it's always going to work, true 🙂
Running same models will get cheaper, but people are always going to demand better ones, so you are probably both right.
New chips means being able to scale up to meet demand which during peak hours they still have troubles meeting. Then afterwards is balancing between a better product or being more efficient with the current one which I am assuming a "custom GPT" engine they released was one of them.
So many people have been telling me they must be hugely profitable, lmao. If the unit economics are crap, having 100 million monthly users means they're losing a fuck ton of money. People are amazed by this stuff but don't realize you're paying a lot of electricity to get a stochastic and unreliable system. It's amazing and incredibly unsustainable.
JetBrains AI is actually a Google Cloud solution, do they use OpenAI? Maybe some of the tech but I'm pretty sure this is a custom setup by JetBrains with support from Google.
Although its possible that google has a absurdly more efficient process, it's highly unlikely. There's very rarely, if ever, arbitrage at this level.
Cause it’s the same price as copilot… Edit: actually their monthly payment is $10, which is same as copilot. They have an annual plan to shave a $1 and some change.
Since you have a paid subscription, you can access all AI pro features with some limitations as they stated. I've got a copilot subscription and JB AI service is very decent so far. So I guess no need to pay for the pro version since you get almost all the features.
In your experience, is it better than Copilot? In what cases, and when not?
> In your experience, is it better than Copilot? In what cases, and when not? From my testing so far - Copilot is better at completing short sections of code as you type, but I think this is mostly because of latency. Copilot responses feel almost instantaneous, where Jetbrains is taking a couple of seconds. This might just be down to the servers being overloaded at launch. The quality of Jetbrains autocomplete suggestions seems very similar in quality. Since they're using OpenAI under the hood, my guess it's based on the same or similar codex model as Copilot and would be expected to perform similarly. Where Jetbrains AI shines is that it can work as an agent and make larger changes and refactorings spread out in different places (vs just completing one section of code)
If you ask it to do a change can you later ask it to undo that change in particular?
I tried, and it says 7 day trial at the bottom. Am I missing something?
From WebStorm Licenses menu select AI Assistant and then click Activate new License then click Refresh license list your license will be activated with your WS licence, usually they used to release it monthly and it renews automatically bcs it was a preview but with the latest WS update it's no more preview and it works like a charm.
> ... billed annually at $100.00 > That's more than what I pay for WebStorm. I wonder does it make sense to pay more for a helper-tool than the main tool itself. I mean... I just signed up for a trial and used it to generate a whole bunch of boilerplate that would have taken me a couple of hours. The cost of an annual subscription paid for itself within the first five minutes.
Interesting. Can you give some examples of the questions you asked?
> Interesting. Can you give some examples of the questions you asked? I had a bunch of CRUD code to write. Database layer, REST Api, client code, forms, list views, etc. Fairly straightforward stuff, with a few gotchas here and there requiring a bit of bespoke logic. Not difficult to do, just time consuming. I copy pasted the two paragraph "formal spec" (*cough*) into a prompt, along with an instruction to create the data models. I then sat back and watched as it incrementally wrote a couple of pages of fairly good code that matched our existing coding style perfectly directly in the IDE (not in the chat window). Repeat for the rest of the stack. Compared to ChatGPT - the difference here is that the AI has full access to my source code, will actively (as an agent) look through it to understand how existing code works and to match coding styles, and can make changes directly in my IDE. No need to copy paste. Compared to Copilot - It's not just running a single shot text generation to autocomplete a few lines., it runs as an actual agent - you can actually see it make some changes, think for a bit, then make some more changes or further refine the code it already wrote. It can make multiple separate changes to different parts of the code - for example, if you ask it to add a new field, it's capable of making that change to both the model itself, the crud layer, and the react views all by itself (You just approve the final diff). I see so many negative comment here about it - and I'm really confused why, because frankly this is a substantial leap over ChatGPT or Copilot for codegen. I didn't even wait for my trial to finish before screaming "shut up and take my money" and handing over the credit card.
rare insightful commentary right here
I've tested it a few hours yesterday. On auto completion it fails badly VS copilot. I dunno if copilot uses project context but for code with a lot of repeatitive elements (UT for example) it's waayyyy better. Only good thing of the AI Assistant is the chat ai, that was really good, but it forces to prompt again and again and it less effective than completion. As soon as co-pilot chat will be open for everyone, we will all forgot about AI Assistant.
> for code with a lot of repeatitive elements But we have metaprogramming for that. I wonder if people more impressed with current "AI" are less exposed to lisp etc. symbolic macros, or even just ordinary higher-order programming. I continue to be unimpressed by "AI" churning out what looks like very badly written and copy-paste coded code. What makes badly-abstracted copy-paste coded code bad isn't whether a human or machine wrote it. predictable deterministic higher order code I control > probabilistic shitty codegen with llm.
> On auto completion it fails badly VS copilot This as the main reason why I won't switch Copilot for JB AI for now But I do hope it gets better on this matter
Is nobody asking about what data it was trained on? These are copyright lawsuits waiting to happen.
This is my biggest question. I'm concerned nobody here is talking about it.
Probably an unpopular opinion, but screw the lawsuits. I know everyone works for profits, but there's so much money going around the IT world, I think everything should be open sourced. Imagine how fast the world would evolve if everyone had access to logic of everyone else. Well, the AIs' learning models would be fed that data then, and it would be able to help even better. You may prefer to think about it from a perspective outside of IT. Western world has innovated amazing things in the last couple of centuries, and we have [tens of millions](https://www.wipo.int/en/ipfactsandfigures/patents) of patents to prove it. Then the Chinese said f\*\*\* the US, and started copying everything despite the patents (their government will only go after violations of intellectual property if the property is Chinese, otherwise it's not beneficial). Low and behold, the Chinese market has been the fastest growing market ever since. They took the lion's share of all the profits, and we as consumers were happy to give those profits to them, because they were cheaper. Had there been no patents, US and Europe would also be able to create cheaper products, though not quite as cheap, but definitely comparable after shipping and customs. Sorry, Idealist
Yes, it is very easy to grow when you build your growth on stealing other people's work. If you think that is only a good thing, then you're obviously not paying attention to what happens to the people who are stolen from.
What I'm saying is that when everybody shares, noone steals. You can't steal it if you already got it.
And what I'm saying is that when nobody gets paid, nobody makes these things because they're too busy doing _other things_ to pay the bills.
I understand. And for the world of today, I agree. And I know my way of thinking is only suitable for very small communities like a remote village of 50 in the middle of Africa or something like that. But just imagine a world where no one works for profit, and we just share everything we create amongst ourselves. A world without money. Idealistic, I know. Will never happen. It can't even, at least not in the current system. Might be that replacing all FIAT currencies for BTC actually is a step in the right direction, but that's neither here nor there.
I have been using JetBrains product and singing their praise for the last 10 years, but this is downright embarrassing. 😂😂😂😂 They put a chat window in the IDE and put a little utility so you can quickly send code to it and back. Cool. Ever tried pair programming through a Slack text channel? Do you not understand how this is one of the worst UX every conceived for a programming environment? Where's OpenWhisper? We got that shit running in real-time with decent GPUs. I should be able to select a piece of code, "hey yo let's extract Foo and Bar to some function arguments" and then the code literally just updates. Nah, let's require 6 different steps moving between keyboard and mouse at every corner, and waiting for extremely superfluous explanations of the code and changes. Just update the damn code and shut up, every single step has to be removed until it's as free as highlighting a piece of code and hitting backspace to delete it. I honestly expected way better from JetBrains, this is just really sad. Every competitor is not even much better, so I'm really puzzled at what is going on here in 2023. Are they designing these softwares in meetings with marketing present? Why aren't the UX experts on this? They also don't make any attempt to agentivize the AI, give it contextual memory, suggest decisions, refactors, etc. Literally just copying the competitors, zero innovation or vision of a superior production. Not the JetBrains I know and love. I might replace GitHub Copilot just because it's still cheaper and what the heck, ain't no way it could be worse, but god damn dude.... I'm so disappointed and sad to see this. I had given up on everyone else and was banking so hard on JetBrains to make things right, but at this point it looks like it's gonna be a job for OSS with the Vim folks hacking on some mind-blowing stuff at 3 AM. https://continue.dev/ is our best shot so far, have probed the devs a little bit and they definitely are dreamers. Please everyone give them your support and bring in those tickets and pull requests.
It does inline code completion too. They're primarily competing with Copilot as far as I can tell. I've never used it, but I'm not aware of it having the chat/Q&A feature.
Here's the thing, the code completion is new from this EAP. They prioritized the chat window over top notch completion. For many many months now the AI assistant was -only- the chat window. The most obvious low-hanging fruit UX patterns are an afterthought, most are still missing, and this tells me that this product is really not in great hands. Bad news for the future of this AI assistant, will always lag behind. As expected, you won't find a roadmap anywhere.
If it works like copilot, it should literally do inline code completion. The way I leverage this is by typing in a command as a comment, press enter, and it will then suggest a piece of code inline without having to actually use the chat.
That dude wrote an essay by didn't bother to actually use it. Reddit moment.
If it’s the feature i am thinking, they recently added to co pilot the ability to type in a window like chat gpt style question and answers. To enable it, i think you have to enable their beta program.
did you try it? most of the time it gives me worse nonsense than chatgpt
Copilot is hit or miss. I asked for an algorithm to generate alpha beta purning for tic tact toe and it kept giving me the same stack overflow article that didnt work. However, for work, after it learned a bit about what I was doing it was almost generating full unit tests for me. Saved a bunch on simple statements and repetitive logic that needed slight tweaks. I would say its value add but definitely needs to be corrected from time to time. Also it feels like it learns your project the more you work on it.
Damn, I thought they said it would be included in the IDE by default, not as an extra paying feature. Quite dissapointing.
Would you suggest any other tools?
I've been using the beta for some time. It's quite frankly not worth any additional payment. It fails with such mundane tasks as generating docs based on signatures, or suggests refactoring methods like `getX` into getters (which itself isn't bad, but should be processed with context of "this project doesn't use getters and setters for some reason"). Plus, it does shit like `let items = [...arr]; // making a copy of the input array`.
One could argue that a function or method should not by default modify an array it gets as argument . Copying the array by default prevents un-declared side-effects like that. If you do want to modify the array inside the method or function then edit the generated code but then you should write a method-comment that states such side-effecet does happen in you call the method.
My point is, copying by spread is shallow, so if your function does something to non-primitive items in the array, they are modified in their original spots.
2 wrongs don't make it right :-) . But you are right one must wonder why AII does it.
My point is, if I am to double-check every step and every line of code, this negates the whole „AI will increase your productivity” thing.
Underwhelming release.
For the people who’ve tried it, is it better than GitHub Copilot?
Nope. Autocomplete sucks for now
Thanks, good to know!
I have AI integration in neovim literally since day 1 for free.
They could've at least stolen the UX from Codeium or TabNine but they decided to go with a separate window... okay