Visual Basic 9.0: Looking Forward

At Microsoft's Professional Developers Conference 2005, Scott Swigart was able to sit down and chat with Microsoft's Amanda Silver, Erik Meijer, Paul Vick, Rob Copeland, Alan Griver, and Jay Roxe about some of the changes coming Visual Basic 9.0. Specifically, VB will have SQL keywords added to the language to support "Language Integrated Query" (LINQ). Here's what the folks working on this feature had to say:

Scott Swigart: To get started, maybe we should go around the table and you can introduce yourselves.

Amanda Silver: I'm a program manager on the compiler, language, and a few other things.

Erik Meijer: I'm an architect on SQL Server, and working with VB on the new stuff.

Paul Vick: I'm a technical lead for the Visual Basic language and compiler.

Rob Copeland: I'm the Product Unit Manager for Visual Basic.

Alan Griver: I'm the Group Manager for Visual Studio Data.

Jay Roxe: I'm the product manager for Visual Basic.

Scott: So, I've seen in the blogs and the press, this thing that you all are working on has been called a lot of different things. I've seen the codename Zorro show up, IQF, LDF, and LINQ. Who would like to describe briefly, what is this? What do these acronyms refer to?

Alan: LINQ stands for "Language Integrated Query." That's the official name for the umbrella project.

Amanda: And, the way to think about LINQ is that it's made up of two parts. One is a set of APIs that are query-enabled, and the second is a set of language extensions made to VB and other .NET languages that allow them to issue queries against those APIs.

Scott: So, what does "Query enabled" mean? Does that manifest itself in VB as a Select keyword in the VB language?

Alan: Right. It's really up to the specific language to decide how they want to implement it, but in VB we're implementing it as SQL-like keywords in the language. So you'll see Select, From, Where, and things like that.

Erik: The key thing is that there's a library of query patterns, operators, or methods, that can capture the basic query operations, and then the .NET languages provide syntactic sugar that their compiler turns into calls into those patterns. This is key because there are several implementations of those patterns. One is over in-memory data structures, this is where we use IEnumerable, but we also implement the same pattern on what we call remote queries, where the query gets sent to, say, SQL Server, and then the results come back. The developer uses exactly the same programming API, and the compiler targets the specific source of the data. There's very deep stuff going on from a programming language point of view.

Scott: In other words, the .NET language exposes data-centric operations, but the actual data source is abstracted, and what's going on underneath might be completely different if you're using Select against a collection of in-memory objects versus a database versus an XML document, but you write the same code?

Rob: Right, you learn one syntax, and then you can query over anything.

Alan: LINQ comes out-of-the-box with support for IEnumerable, for instance, and then there are two other APIs that people will start hearing about. One is called DLinq, that's built on top of ADO.NET, and the other is XLinq, which is for querying XML.

Scott: People have talked about this relational-object-XML (ROX) problem for a long time. In the first cut of this, how much of the ROX problem space are you addressing with this project?

Erik: When you're talking about the ROX triangle, if you look at a language like SQL, it has queries, and it has a data model of collections of tables of rows. It you look at XQuery, it has a query syntax, and a data model. It has FLWR expressions, and an XML node list. With in-memory objects, you have a data model in memory, as in collections, but no query syntax. So each point on the triangle has to handle querying and data.

What we've done is separate the query syntax from the data model, so the queries work over any collection. It doesn't matter what the elements in the data collections are, whether they're relational rows, or XML nodes, or objects in memory. Separating the data model from the query model is how we solve the ROX problem.

Scott: With DLinq, you're using something that's built on top of ADO.NET. If you're a VB developer, how much of ADO.NET are you still seeing in your code? Are you still using DataSets, Connections, and DataAdapters? Or, are you using something at a higher level, and those objects are being used behind the scenes?

Paul: Essentially, what DLinq does is let you model your data as objects. And then you interact with those objects most of the time. You can still interact with the lower level ADO.NET objects if you need to, but the power of DLinq is that instead of dealing with a DataSet that has a Customers data table, you're just dealing with Customer objects.

Scott: So you'd have a Customer class with properties that map to the columns in the Customers table in the database? And then when you do your Select, which is now a VB language keyword, what you get back is a collection of Customer objects populated with data from the database?

Paul: Exactly.

Scott: So you have something that's very strongly typed, and very closely mapped to the database. Are there tools then that will look at the database and generate your classes for you?

Paul: Yes.

Erik: Yes. Or, you can code it yourself and use attributes to specify the mapping. Right now DLinq keeps it very simple. You can map tables, you can map relationships, and you can change names so that a column appears in the database with one name, and in your code with a different name.

Paul: The really cool part is, the way that LINQ is designed, you feel like you're working with local objects, but if you query on top of them, you're not filtering the local objects. The query gets remoted back to the database.

Scott: This is really what everybody wanted. People didn't necessarily want Recordsets or DataSets, they wanted Customers or Orders, but you needed a generic way to talk to the database because it could contain anything. Typed Datasets were a step in this direction, where it pulled in the meta-data and made it look like you were working with classes, but underneath it was just wrapping the generic Dataset. But, you're saying LINQ really does let you just work with simple lightweight classes. What happens when you make changes to your classes? How do changes get propagated back to the database?

Erik: So let me say something about Typed Datasets, because I think there is a sort of Grand Canyon there. Once you have the data, it is strongly typed, but the queries are still done as strings, so you don't have a way to insure that the results that you get back are in the shape that you specified in your Typed Dataset. But with LINQ, your queries are written in VB or C#, and the compiler knows about the structure of the data, so you know that the results of the query will match your strong typing.

Scott: One of the big advantages of strong typing is compile time checking, but also things like IntelliSense, so I'm guessing that one of the benefits of LINQ is that column names will pop up in IntelliSense when you're writing queries.

Amanda: Exactly.

Paul: The interesting thing is that you're saying that DLinq is replacing DataSets, but you shouldn't think of it that way. If you force strong typing, you lose the dynamic capabilities. There's still a place for DataSets, when you're dealing with dynamic data, data where you can't know the shape at design time.

Scott: Right, the scenario where the user of an application might have the ability to add custom columns, or something like that.

Paul: Right, so it's not an either/or, but DLinq is an addition onto what you have. It depends on the nature of the work that you're doing whether you want the data with strong or loose typing.

Alan: And in VB, one thing we're looking at is the concept of dynamic interfaces, which gives you an early bound feel, to this late bound data.

Amanda: Dynamic interfaces would let you create a sort of facade that's a clue to the compiler to tell it what IntelliSense it should show, what members you expect to be able to bind to at run time.

Scott: Conceptually, that sounds almost like the DataSet/Typed DataSet kind of model, where you are wrapping a generic object, the DataSet, in something that gives it strong typing, but underneath, it's not the object that's enforcing type checking. The interface to the object, the wrapper, is enforcing the type checking.

Alan: And the power of that is often when you're writing code, there are pieces of the data that you know have to be there, and some of your code works specifically against that data, but everything else that's added on at the end, that stuff can change. New columns can be added, and you don't have to worry about that for the piece of code your writing. Your code can be strongly typed, even though not everything is strongly typed.

Erik: And for the real geeks....

Amanda: And Erik is the one to talk here.

Scott: So just to be clear, this is Erik Meijer, speaking on behalf of all Real Geeks.

Erik: What these dynamic interfaces let you do is something that you cannot express in traditional type systems. Suppose I know that the result I get will have a Name property of type string. But for the rest of the item, it doesn't implement any interface that describes it. What you can do with dynamic interfaces is define the interface to say that the item will have a Name property, and then you can impose that interface on different types. There's no way you could define that in a traditional type system.

Scott: It sounds like the dynamic interfaces feature would have value outside of just LINQ. This sounds like its own "Big Language Feature".

Amanda: Yes, some of the customers we've been talking to have been telling us that it's very compelling for data binding-like scenarios, but without using the existing data binding mechanisms.

Paul: We've sort of wandered off of LINQ, but if you look at what dynamic interfaces gives you, we talked about that chasm where on one side you have strong typing, and you get IntelliSense. At the other end, you have loose typing, like Object, or Dataset, but you get no IntelliSense or compile time checking. Right now, you have to pick your poison. We're looking at ways to dump dirt into the chasm and make so that you can move smoothly from one end of the spectrum to the other, and you don't suddenly loose all the benefits of strong or loose typing.

Amanda: I definitely wouldn't say that we're done here either. There's a huge amount of stuff that we could do in the tools and the language, and this is just another step in making VB a really comfortable language and type system to program in.

Erik: This is kind of my pet horse.

Paul: Uh, your hobby horse?
(laugher)

Erik: Right, hobby horse, that's the way it goes with us non-native speakers....

If you look at a lot of scripting languages, they're all dynamically typed, and if you want to add strong typing, you're stuck, because they don't have any notion of static types. Then, if you look at other languages, like Java or C#, they have no notion at all of dynamic types. You have to use reflection, but then you are going outside of the language.

VB, in that sense, is a very unique language in that it allows both. It makes it easy to turn the dial, and say that you want to be late bound, or early bound, or somewhere in between. I don't know of any other mainstream languages or even research languages that have that capability. We're looking at making that even better, so as a developer you can just stay where it's most convenient for solving your problem.

I wish I hadn't taken so long to really look into VB. From a language perspective, it's really amazing.

Paul: Dynamic interfaces are the first step. It is our goal to make VB even more dynamic, but not lose any of the static typing so that VB can be the best language it can be, and be one of the only languages that straddles these worlds, and does a really good job at it.

Scott: That's one of the things I've noticed. With some of the new stuff your doing, it looks and feels like you're programming in a dynamic language, but then when you hit F5, the compiler is able to check tons of stuff that a typical dynamic/scripting language wouldn't, and give you compile errors instead of runtime errors.

That leads me to another question. You have this basic LINQ functionality, but then languages like C# and VB can choose how they want to implement it. On the C# side, it seems like you hear a lot about lambda functions and anonymous types, and things that make you feel like you have a doctorate in computer language theory, but I'm guessing that from the VB side, you're not going to force VB developers to go Google "lambda functions" before they can use LINQ effectively. When you implement this in VB, what are your prime directives for how you provide LINQ to the developer.

Paul: That's the question that comes up a lot. When you sit down with VB, you're not going to have to know about these language theory topics. That's why we're using a very SQL-like syntax, because it's something that's very approachable and understandable, and the great thing is that the SQL syntax in VB is built on top of all the rocket science. The advanced stuff will be available to you if you need it, but the focus is on making it available with straight-forward syntax and paradigms that you already know. We feel like, if it's a success, when you use these features, it should feel like a very natural extension of what you already know.

Amanda: Another thing that we always keep in the back of our mind is discoverability, because the whole nature of using the tool is all about trying something out. If you read about something that's interesting, like a class library, and want to see how it works, the best way is to paste in code, and call some of its methods, and see if it actually does what you expected it to. With all of these language additions, we want to add on to that discoverability so that they're easy to find and use to create the program that you see with your mind's eye.

Scott: So if you wanted to write your own user defined aggregate, for example, you might need to know about something more advanced, but to use Select, Insert, Update, Delete, Where, Order By, Group By....

Amanda: Right, right, so the discoverability of these features is a much more sequential act. You can get a result back from a query, and you get IntelliSense on that result, and you can also query over the results. The feeling is, it works exactly how you'd expect it to work.

Alan: And that's actually one of the key things about LINQ, since the underlying sub-strata is the same across VB and C#, it becomes very easy to pass data back and forth. You're not loosing the core of what .NET is, which is multi-language.

Erik: The key of good language design is to make sure that all the rocket science works for you, without being in your face. You want those rocket engines powering the language, but you don't want to have to know all about them to use the language.

Scott: It seems like the stakes are so high when you're talking about the language. If you release a bad API, well, you can just come out with a new API and say, "Don't use the old one." But if you screw up the language, you're kind of stuck.

Paul: VB has been around long enough that we've lived through decisions that you look back on and think, "I wonder what they were thinking when they came up with that."

Amanda: You mean, "What was I thinking when I came up with that?"
(laugher)

Scott: So if I want an "Ain't" keyword in VB, you're saying I'm going to have to lobby pretty hard?

Paul: That's right.
(laugher)

Scott: So when will we get it? Are you targeting a specific release, or is it more, "It will ship when it's ready."

Amanda: It will definitely ship when it's ready, but one thing to take away from all this, this interview, and the PDC, is that we really want to get it in people's hands early. We want a lot of customer feedback, because it's such a big project, we expect it to have a lot of bake time.

Alan: We haven't announced a ship schedule, but we are providing bits that go live on MSDN this week, and the bits install right on top of VS 2005 Beta 2 or RC, so people can play with it now.

Scott: One of the things about querying is that it's more than just the query operators. Products like SQL Server protect you from how you order your joins and things like that. There's a query optimizer that looks at that kind of stuff, and switches thing around to make the query run faster. Is there a query optimizer with LINQ, or in some cases, might you have to know what you're doing to get the best performance?

Alan: With things like DLinq, we're relying on SQL Server's optimizer. For other things, it's one of the things on a list of many things we're looking at.

Paul: It's definitely something that comes up.

Scott: For objects in memory, sometimes, there's only so far you can go wrong, but in other instances, there might be an order of magnitude performance difference with optimization.

Erik: The beauty of how we're designing the infrastructure is that it allows you to build things like query optimization. You'll be able to use exactly the same infrastructure, and exactly the same query syntax. This lets the underlying implementation get smarter over time, and do better with exactly the same query.

Paul: The design very carefully doesn't lock you in to a specific implementation. That's different than something like SQL Server, where you get the SQL Server query optimizer, and that's it, and if it doesn't do what you want, it's sort of like, "that's tough." LINQ is a flexible architecture in that way.

Scott: When we talk about DLinq, because it's built on ADO.NET, I'm guessing there's no limitation in what the back-end database is, so this would work whether it's Oracle, or SQL Server, or Access, or an AS400? Or, do you get a better database experience with certain databases?

Alan: DLinq today works against SQL Server.

Paul: That's because basically, DLinq has to build queries, SQL statements, that get sent to the back-end database.

Scott: Oh, so DLinq needs to understand the exact SQL syntax for the thing it's talking to.

Paul: It's sort of the equivalent to the provider model that we today.

Scott: So there's nothing that would stop Microsoft, or anyone else, from building database specific APIs, and those APIs would compose the SQL that a specific database needed so that you could use LINQ to query it just like anything else?

Alan: That's right.

Scott: And, you'd need to know lambda functions to do this.
(laugher)

Erik: Well, now you're talking about lower level stuff than just using the query syntax.

Paul: I was thinking about this earlier, and it's very equivalent to generics, where it's very easy to use a generics collection, but it's not trivial to go build a generics collection.

Scott: What are some of the things that you would like people to know, especially as it relates to the VB implementation? If there are a few main themes that you don't want to get lost in the noise, what would they be?

Amanda: One of the things is the dynamic interfaces concept that we talked about. It's something that we're very excited about. That's part of making the Visual Basic type system span both the static and dynamic typing worlds, and that's also shown in the nullable design we're proposing. And also there's the deep XML integration in VB, but I'll let Erik talk more about that.

Erik: Who, me?

If you talk about the experience, what we did for VB was to add deep XML support into the language, and also into the tool. Let me talk about the language first. One of the parts of LINQ is XLinq which is a new XML API. But what you really want to do is cut and paste some XML into your editor and work from that. And if you first need to write a bunch of constructor calls to the XML API, that doesn't really work. What we did is, in VB, we support XML literals. So you could save a spreadsheet as XML, and just cut and paste that into your program.

Scott: VB is an imperative language, and XML is declarative, and what you're saying is that now in VB you can do both? You can just throw the XML right into the middle of your code and populate the elements with data?

Erik: Right. And what we're doing for tools support is your code editor is a full XML editor that helps you work with the XML. It's smart about tags, so it will automatically close all your tags for you, and format everything.

Scott: So then the compiler can also make sure that this XML that you just pasted in is well formed?

Erik: Exactly.

Alan: This is really key for typical business scenarios also. You don't often get XSDs from the business partners that you're working with, but they will send you an example XML document and say, "Make your XML look like this." There's nothing easier, than copying and pasting that XML into your code, having it be verified for you, and then saying, 'Replace here where they have an example first name, with an actual database call." Now you have a much quicker way of developing these multi-partner XML based applications.

Amanda: I just wanted to say that the process of copying and pasting XML doesn't prevent you from also schematizing that XML. If you have a schema, you can use that to have the tool provide you better IntelliSense.

Erik: This also fits into this whole idea of bridging the gap between static and dynamic data. If you have a schema, we want to take full advantage of that, by providing IntelliSense, or drop-downs as you type the XML. If you don't have the schema, we don't want to prevent you from working with the XML, but if you do, you should get better support in the tools, better error checking.

Scott: Which is very VB-like. You're saying, "Here's sort of a quick, easy way to work with XML, but if you're using XML in a more advanced way, and you're using schemas, there's support for that too."

Amanda: I think that's one of the things. Schematizing your XML shouldn't be an "advanced thing" any more. It should be something that you can just add to the tool to give the tool a way to give you more feedback about whether you're writing the right code. So we want to use these "advanced" concepts, but use them in a way that's very obvious.

Paul: I just want to come back to the question of, "What do we hope people walk away with?" We've talked about some low level stuff, but the high-level message is that there's really some interesting things happening with the Visual Basic language. We've spent the last couple versions, sort of absorbing the massive amount of stuff that's in the .NET Framework. Now that we've accomplished that mission, we're focusing on adding interesting things that will be unique to the language, and that we're in a uniquely good position to do. I hope that people really get a feeling of that.

Alan: The other thing I wanted to point out is that as people look at MSDN, which has the C# flavor of LINQ and the VB flavor of LINQ, that it really pays, especially early on, to look at both, because with the different teams focusing on different areas early on, and you'll get a better view of the whole feature.

Amanda: For example, we've been working on XML integration pretty fiercely, meanwhile, we're don't have support in this initial version for DLinq.

Rob: By "initial version", we mean the bits that we're giving out at the PDC.

Alan: Right. VB will have full support for DLinq.

Scott: What, you don't want headlines of "VB won't LINQ to databases." It's not like people ever publish myths about supposed VB limitations, right?

Rob: Exactly.

Scott: One of the last things I want to talk about is transactions. That's a core database concept, what does that look like in LINQ?

Amanda: I think we have a good set of functionality today, and there's even more we can do to make that even easier.

Scott: So one would expect that with DLinq, you get transactions, but because you can now join your business objects with your database, and with XML, you might want to make changes and then rollback. In the first version that ships, will we see that, or will it be that databases understand transactions so they'll get it, but objects don't understand transactions, so they won't get transaction support:

Paul: I think it's an open question at this point. It's like the question of query optimization. It's an important question that comes up very quickly when you start talking about using this. We recognize that it's an important scenario, but a lot of work remains to be done.

Amanda: That's one of the benefits of getting it out early is that our customers can tell us if there's this stack of 10 or 15 things that we have to do before they can use it at all. What do we have to get done to make this compelling to them?

Erik: When you talk about transactions, you're touching something that's very, very dear to my heart.

Scott: Another pet horse?

Erik: Exactly. Because the problem is, if you have a transaction in the database, but you're also doing in memory things, you can roll back the database, but your in memory state is messed up. You want to roll that back too. Recently, there's been a lot of people looking at software transactions. If someone can predict the future and say, "The world will ever only need five computers," then I can predict the future and say, "We will use transactions everywhere; in-memory, database, XML, everywhere. We will have one way for dealing with concurrency. "

Scott: While it hasn't been an intrinsic language feature, it seems like something you do a transactions a lot, where something fails and you want to roll back. You sort of want "transaction oriented programming."

Erik: You talk about making things easier for the VB developer. The database developer uses transactions a lot because it makes it very easy to reason about your application. Either the transaction succeeds, and all the state changes take place, or the transaction fails, and nothing has happened. Even without database integration, it could bring that simplicity to normal programming. You can do something and it either succeeds, or nothing has happened, and I think that's an extremely powerful mental model for programming.

Paul: As we think ahead to multi-core processors coming more on-line, concurrency is something that more programmers will have to deal with. It's a fertile ground for work at the moment.

Scott: So once you solve the data stuff, you're going to make threads super easy?
(laugher)

Paul: Exactly, I mean how hard can it be, it's not like anyone's been working on it for 30 years.

For More Information

Scott Swigart is consultant specializing in convergence of current and emerging technologies. Scott can be contacted at http://www.swigartconsulting.com/.