SQL dialect
Spark SQL SQL Parser for Java and .NET
General SQL Parser (GSP) parses Spark SQL with a dedicated, hand-tuned grammar — full AST access, offline syntax validation, formatting, rewriting, and column-level data lineage , including complete procedural SQL support. One commercial SDK, 38 dialects, Java and .NET.
Example
Spark SQL SQL that GSP parses
CACHE TABLE sales_cache AS
SELECT region, SUM(amt) AS total
FROM sales WHERE year = 2026
GROUP BY region;Parse it in Java
import gudusoft.gsqlparser.*;
TGSqlParser parser = new TGSqlParser(EDbVendor.dbvsparksql);
parser.sqltext = sql; // the Spark SQL SQL above
if (parser.parse() == 0) {
System.out.println(parser.sqlstatements.size() + " statement(s) parsed");
} else {
System.out.println(parser.getErrormessage());
}The .NET edition exposes the same API in C# — see the C# quick start.
Coverage
What GSP handles in Spark SQL
- CACHE TABLE
- REFRESH TABLE
- ADD JAR/FILE resources
- explode(from_json(...)) lineage
- FOR and LOOP statements
- Procedural SQL: SQL scripting control flow: FOR loops and LOOP/END LOOP statements
Beyond parsing, the same AST powers table/column extraction, validation, formatting, and column-level lineage for Spark SQL.
Actively maintained
Recent Spark SQL parser updates
- 2026-06-02 [SparkSQL] Parse PRIMARY KEY / FOREIGN KEY in CREATE TABLE
- 2026-05-29 [Hive, SparkSQL] Structured-dataflow lineage for explode(from_json(...))
- 2026-03-03 [SparkSQL] Support FOR loop statement
Full history in the release notes.
Go deeper
Spark SQL resources
- Spark SQL syntax reference — dialect documentation
- Spark SQL keyword compatibility — every keyword, and whether it can be an identifier
- Java quick start — Maven/Gradle setup and first parse
- C# quick start — NuGet setup for .NET
Common questions
Does GSP parse Spark SQL stored procedures and procedural SQL?
Yes. GSP parses SQL scripting control flow: FOR loops and LOOP/END LOOP statements into a full AST — procedure bodies become real statement trees you can traverse, not opaque text blocks. This is what makes lineage and impact analysis work inside procedural code.
Do I need a Spark SQL database connection to validate SQL?
No. GSP validates Spark SQL syntax completely offline. You get error line and column positions, the offending token, and a hint — with no server, driver, or credentials involved.
How complete is the Spark SQL grammar?
GSP uses a dedicated hand-tuned grammar for Spark SQL — not a generic SQL grammar with flags. The parser recognizes 330 Spark SQL keywords and knows which 313 of them can also be used as identifiers, which is exactly the kind of edge case that breaks generic parsers.
Can I use the Spark SQL parser from C# / .NET?
Yes. The .NET edition (NuGet package gudusoft.gsqlparser, .NET Standard 2.0) exposes the same API surface as the Java edition, so every Spark SQL parsing feature on this page works in C# and VB.NET.
Can GSP extract column-level lineage from Spark SQL SQL?
Yes. The built-in DataFlowAnalyzer produces column-level lineage, impact analysis, and call graphs from Spark SQL scripts — it is the engine behind Gudu SQLFlow and the lineage integrations for DataHub and OpenMetadata.