Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrate sqlparser #361

Closed
wants to merge 39 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
2a2fd87
Fix String escaping and add test
mqus Jun 2, 2020
4f1d20d
Switch to raw strings
mqus Jun 4, 2020
a174323
Add tests for different quotes
mqus Jun 12, 2020
620cd9a
fix analyzer error
mqus Jun 12, 2020
9416a18
Start to integrate sqlparser
mqus May 6, 2020
7ec04ce
temporary state
mqus May 26, 2020
34ceedb
complete initial queryparsing
mqus Jun 2, 2020
fc5a2c2
Start to integrate the analyzer into the qmprocessor
mqus Jun 13, 2020
3d859a9
Add some todo comments
mqus Jun 14, 2020
94aeac2
Finish alpha-grade version of the sqlparser integration
mqus Jun 15, 2020
73b7192
cleanup a bit, regenerate example code, add more todos
mqus Jun 16, 2020
0efaecd
Add quoting tests, use list for varlist positions, quickfix tests
mqus Jun 17, 2020
bbb846a
Update sqlparser, fix tests (for now)
mqus Jun 19, 2020
513eabf
fmt
mqus Jun 19, 2020
956537a
fix linter issues
mqus Jun 19, 2020
09d50fe
refactor query analysis, use toLiteral for literal strings
mqus Jun 20, 2020
804cd78
Add missing assertions, nicer error messages with span highlights
mqus Jun 21, 2020
5792c69
Readme: remove outdated limitations on streams and databaseviews and …
mqus Jun 21, 2020
6923120
plan generator unit tests, add mini-features from todos
mqus Jun 22, 2020
523211d
Refactor testing and add new view tests, update sqlparser to latest r…
mqus Jul 8, 2020
8bb6fbc
Fix an error message, finish refactoring View conversion
mqus Jul 8, 2020
9c49c9e
Introduce return type checker and make sure that generated test sourc…
mqus Jul 9, 2020
431c996
Add primitive type mapper, add tests, fix bug
mqus Jul 11, 2020
f920a07
Clean up unnecessary dependencies: AnalyzerEngine in Queryable/Entity…
mqus Jul 11, 2020
3897bf9
Add QueryAdapter unit tests
mqus Jul 12, 2020
0af2a9c
Complete Queryparser unit tests
mqus Jul 12, 2020
be29dc0
Remove converter and unnecessary table constraint conversion
mqus Jul 12, 2020
352d75b
Clean up return type checking and add some tests (incomplete)
mqus Jul 12, 2020
7b7b732
Add some integration tests and update generated code in example
mqus Jul 29, 2020
98df943
Extend QueryAdapter docs on noReturnQuery and add test idea to integr…
mqus Jul 29, 2020
c802294
Clean up files, add @nonNull, add docstrings. Update sqlparser for bu…
mqus Jul 29, 2020
7da688e
Add integration test for bool serialization and unit test for primiti…
mqus Jul 30, 2020
6634f3b
Add more unit tests and integration tests
mqus Jul 30, 2020
8279bbe
Merge remote-tracking branch 'upstream/develop' into integrate-sqlparser
mqus Jul 31, 2020
2c3d326
Merge remote-tracking branch 'upstream/develop' into integrate-sqlparser
mqus Jul 31, 2020
6d8cc21
integrate WITHOUT ROWID
mqus Jul 31, 2020
9925320
Apply quick fixes to small review issues
mqus Aug 12, 2020
1d2a210
Refactor dependencies to only propagate sql names of entities, fix co…
mqus Aug 13, 2020
6ac8111
Add comment to pinned analyzer version and add more tests to daowriter
mqus Aug 15, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 9 additions & 13 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -396,11 +396,6 @@ You can then query the view via a DAO function like an entity.

It is possible for DatabaseViews to inherit common fields from a base class, just like in entities.

#### Limitations
- It is now possible to return a `Stream` object from a DAO method which queries a database view. But it will fire on **any**
`@update`, `@insert`, `@delete` events in the whole database, which can get quite taxing on the runtime. Please add it only if you know what you are doing!
This is mostly due to the complexity of detecting which entities are involved in a database view.

## Data Access Objects
These components are responsible for managing access to the underlying SQLite database and are defined as abstract classes with method signatures and query statements.
DAO classes can use inherited methods by implementing and extending classes while also using mixins.
Expand All @@ -421,9 +416,8 @@ abstract class PersonDao {

### Queries
Method signatures turn into query methods by adding the `@Query()` annotation with the query in parenthesis to them.
Be patient about the correctness of your SQL statements.
They are only partly validated while generating the code.
These queries have to return either a `Future` or a `Stream` of an entity or `void`.
Your SQL queries will be validated completely while generating the code.
These queries have to return either a `Future` or a `Stream` of an entity, a view, a primitive value or `void`.
Comment on lines +419 to +420
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❔ The query parser, in the future, will allows us to parse @Query results into non-primitive values, right? #94

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will need another PR but it should be as simple as adding another Queryable subtype and using the same processing as for entities/views. The type checking and the mapping already exists.

Returning `Future<void>` comes in handy whenever you want to delete the full content of a table, for instance.
Some query method examples can be seen in the following.

Expand Down Expand Up @@ -460,6 +454,13 @@ final name = '%foo%';
await dao.findPersonsWithNamesLike(name);
```

#### Limitations
- Supplying more than one query (separated by a semicolon) will not work.
Please use transactions or successive calls to DAO-Methods instead.
- The underlying frameworks only support up to 999 different parameters. This
seems like a big number, but each element of a list parameter counts towards it
and the length of these lists could change at runtime.

### Data Changes
Use the `@insert`, `@update` and `@delete` annotations for inserting and changing persistent data.
All these methods accept single or multiple entity instances.
Expand Down Expand Up @@ -520,11 +521,6 @@ StreamBuilder<List<Person>>(
```

#### Limitations
- Only methods annotated with `@insert`, `@update` and `@delete` trigger `Stream` emissions.
Inserting data by using the `@Query()` annotation doesn't.
- It is now possible to return a `Stream` if the function queries a database view. But it will fire on **any**
`@update`, `@insert`, `@delete` events in the whole database, which can get quite taxing on the runtime. Please add it only if you know what you are doing!
This is mostly due to the complexity of detecting which entities are involved in a database view.
- Functions returning a stream of single items such as `Stream<Person>` do not emit when there is no query result.

### Transactions
Expand Down
6 changes: 3 additions & 3 deletions example/lib/database.g.dart

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

22 changes: 9 additions & 13 deletions floor/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -396,11 +396,6 @@ You can then query the view via a DAO function like an entity.

It is possible for DatabaseViews to inherit common fields from a base class, just like in entities.

#### Limitations
- It is now possible to return a `Stream` object from a DAO method which queries a database view. But it will fire on **any**
`@update`, `@insert`, `@delete` events in the whole database, which can get quite taxing on the runtime. Please add it only if you know what you are doing!
This is mostly due to the complexity of detecting which entities are involved in a database view.

## Data Access Objects
These components are responsible for managing access to the underlying SQLite database and are defined as abstract classes with method signatures and query statements.
DAO classes can use inherited methods by implementing and extending classes while also using mixins.
Expand All @@ -421,9 +416,8 @@ abstract class PersonDao {

### Queries
Method signatures turn into query methods by adding the `@Query()` annotation with the query in parenthesis to them.
Be patient about the correctness of your SQL statements.
They are only partly validated while generating the code.
These queries have to return either a `Future` or a `Stream` of an entity or `void`.
Your SQL queries will be validated completely while generating the code.
These queries have to return either a `Future` or a `Stream` of an entity, a view, a primitive value or `void`.
Returning `Future<void>` comes in handy whenever you want to delete the full content of a table, for instance.
Some query method examples can be seen in the following.

Expand Down Expand Up @@ -460,6 +454,13 @@ final name = '%foo%';
await dao.findPersonsWithNamesLike(name);
```

#### Limitations
- Supplying more than one query (separated by a semicolon) will not work.
Please use transactions or successive calls to DAO-Methods instead.
- The underlying frameworks only support up to 999 different parameters. This
seems like a big number, but each element of a list parameter counts towards it
and the length of these lists could change at runtime.

### Data Changes
Use the `@insert`, `@update` and `@delete` annotations for inserting and changing persistent data.
All these methods accept single or multiple entity instances.
Expand Down Expand Up @@ -520,11 +521,6 @@ StreamBuilder<List<Person>>(
```

#### Limitations
- Only methods annotated with `@insert`, `@update` and `@delete` trigger `Stream` emissions.
Inserting data by using the `@Query()` annotation doesn't.
- It is now possible to return a `Stream` if the function queries a database view. But it will fire on **any**
`@update`, `@insert`, `@delete` events in the whole database, which can get quite taxing on the runtime. Please add it only if you know what you are doing!
This is mostly due to the complexity of detecting which entities are involved in a database view.
- Functions returning a stream of single items such as `Stream<Person>` do not emit when there is no query result.

### Transactions
Expand Down
45 changes: 24 additions & 21 deletions floor/lib/src/adapter/query_adapter.dart
Original file line number Diff line number Diff line change
Expand Up @@ -39,25 +39,32 @@ class QueryAdapter {
@required final T Function(Map<String, dynamic>) mapper,
}) async {
final rows = await _database.rawQuery(sql, arguments);
return rows.map((row) => mapper(row)).toList();

return rows.map(mapper).toList();
}

/// Executes a SQLite query that does not return any values.
/// It will also trigger the [_changeListener] of affected entities if this
/// query is expected to change something.
Future<void> queryNoReturn(
final String sql, {
final List<dynamic> arguments,
final Set<String> changedEntities,
}) async {
// TODO #94 differentiate between different query kinds (select, update, delete, insert)
// this enables to notify the observers
// also requires extracting the table name :(
await _database.rawQuery(sql, arguments);

if (_changeListener != null &&
changedEntities != null &&
changedEntities.isNotEmpty) {
changedEntities.forEach(_changeListener.add);
}
}

/// Executes a SQLite query that returns a stream of single query results.
Stream<T> queryStream<T>(
final String sql, {
final List<dynamic> arguments,
@required final String queryableName,
@required final bool isView,
@required final Set<String> dependencies,
@required final T Function(Map<String, dynamic>) mapper,
}) {
assert(_changeListener != null);
Expand All @@ -71,14 +78,12 @@ class QueryAdapter {

controller.onListen = () async => executeQueryAndNotifyController();

// listen on all updates if the stream is on a view, only listen to the
// name of the table if the stream is on a entity.
// listen on all updates where the updated table
// is one of the dependencies of this query.
final subscription = _changeListener.stream
.where((updatedTable) => updatedTable == queryableName || isView)
.listen(
(_) async => executeQueryAndNotifyController(),
onDone: () => controller.close(),
);
.where(dependencies.contains)
.listen((_) async => executeQueryAndNotifyController(),
onDone: () => controller.close());

controller.onCancel = () => subscription.cancel();

Expand All @@ -89,8 +94,7 @@ class QueryAdapter {
Stream<List<T>> queryListStream<T>(
final String sql, {
final List<dynamic> arguments,
@required final String queryableName,
@required final bool isView,
@required final Set<String> dependencies,
@required final T Function(Map<String, dynamic>) mapper,
}) {
assert(_changeListener != null);
Expand All @@ -104,13 +108,12 @@ class QueryAdapter {

controller.onListen = () async => executeQueryAndNotifyController();

// Views listen on all events, Entities only on events that changed the same entity.
// listen on all updates where the updated table
// is one of the dependencies of this query.
final subscription = _changeListener.stream
.where((updatedTable) => isView || updatedTable == queryableName)
.listen(
(_) async => executeQueryAndNotifyController(),
onDone: () => controller.close(),
);
.where(dependencies.contains)
.listen((_) async => executeQueryAndNotifyController(),
onDone: () => controller.close());

controller.onCancel = () => subscription.cancel();

Expand Down
75 changes: 65 additions & 10 deletions floor/test/adapter/query_adapter_test.dart
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,33 @@ void main() {

verify(mockDatabaseExecutor.rawQuery(sql, arguments));
});

test('executes query with update', () async {
final streamController = StreamController<String>();
const entityName = 'person';

final underTest = QueryAdapter(mockDatabaseExecutor, streamController);

final arguments = [123];
await underTest.queryNoReturn(sql,
arguments: arguments, changedEntities: {entityName});

expect(streamController.stream, emits(entityName));
verify(mockDatabaseExecutor.rawQuery(sql, arguments));

await streamController.close();
});

test('executes query with update with no _changeListener present',
() async {
const entityName = 'person';

final arguments = [123];
await underTest.queryNoReturn(sql,
arguments: arguments, changedEntities: {entityName});

verify(mockDatabaseExecutor.rawQuery(sql, arguments));
});
});
});

Expand Down Expand Up @@ -162,7 +189,7 @@ void main() {
when(mockDatabaseExecutor.rawQuery(sql)).thenAnswer((_) => queryResult);

final actual = underTest.queryStream(sql,
queryableName: entityName, isView: false, mapper: mapper);
dependencies: {entityName}, mapper: mapper);

expect(actual, emits(person));
});
Expand All @@ -179,8 +206,7 @@ void main() {
final actual = underTest.queryStream(
sql,
arguments: arguments,
queryableName: entityName,
isView: false,
dependencies: {entityName},
mapper: mapper,
);

Expand All @@ -195,7 +221,7 @@ void main() {
when(mockDatabaseExecutor.rawQuery(sql)).thenAnswer((_) => queryResult);

final actual = underTest.queryStream(sql,
queryableName: entityName, isView: false, mapper: mapper);
dependencies: {entityName}, mapper: mapper);
streamController.add(entityName);

expect(actual, emitsInOrder(<Person>[person, person]));
Expand All @@ -211,7 +237,7 @@ void main() {
when(mockDatabaseExecutor.rawQuery(sql)).thenAnswer((_) => queryResult);

final actual = underTest.queryListStream(sql,
queryableName: entityName, isView: false, mapper: mapper);
dependencies: {entityName}, mapper: mapper);

expect(actual, emits([person, person2]));
});
Expand All @@ -230,8 +256,7 @@ void main() {
final actual = underTest.queryListStream(
sql,
arguments: arguments,
queryableName: entityName,
isView: false,
dependencies: {entityName},
mapper: mapper,
);

Expand All @@ -248,7 +273,7 @@ void main() {
when(mockDatabaseExecutor.rawQuery(sql)).thenAnswer((_) => queryResult);

final actual = underTest.queryListStream(sql,
queryableName: entityName, isView: false, mapper: mapper);
dependencies: {entityName}, mapper: mapper);
streamController.add(entityName);

expect(
Expand All @@ -262,6 +287,7 @@ void main() {

test('query stream from view with same and different triggering entity',
() async {
const otherEntity = 'otherEntity';
final person = Person(1, 'Frank');
final person2 = Person(2, 'Peter');
final queryResult = Future(() => [
Expand All @@ -271,7 +297,7 @@ void main() {
when(mockDatabaseExecutor.rawQuery(sql)).thenAnswer((_) => queryResult);

final actual = underTest.queryListStream(sql,
queryableName: entityName, isView: true, mapper: mapper);
dependencies: {entityName, otherEntity}, mapper: mapper);
expect(
actual,
emitsInOrder(<List<Person>>[
Expand All @@ -282,7 +308,36 @@ void main() {
);

streamController.add(entityName);
streamController.add('otherEntity');
streamController.add(otherEntity);
});

test('unrelated update should not update stream', () async {
const otherEntity = 'otherEntity';
final person = Person(1, 'Frank');
final person2 = Person(2, 'Peter');
final queryResult = Future(() => [
<String, dynamic>{'id': person.id, 'name': person.name},
<String, dynamic>{'id': person2.id, 'name': person2.name},
]);
when(mockDatabaseExecutor.rawQuery(sql)).thenAnswer((_) => queryResult);

final actual = underTest.queryListStream(sql,
dependencies: {entityName}, mapper: mapper);
//first, emit the result
expect(
actual,
emitsInOrder(<List<Person>>[
[person, person2],
]));

//after emitting the first result, no other updates were made
expect(actual.skip(1), emitsDone);

streamController.add(otherEntity);
streamController.add(otherEntity);

await Future<void>.delayed(const Duration(milliseconds: 100));
await streamController.close();
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 We can remove this line as we close StreamControllers in the tearDown block after each test.

Suggested change
await streamController.close();

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I remove that, the test will fail because of a deadlock/timeout, since the emitsInOrder still waits for the Stream to complete, but the Stream only gets closed after the test succeeded.

});
});
}
24 changes: 24 additions & 0 deletions floor/test/integration/boolean_conversions/bool_test.dart
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,24 @@ void main() {
final actual = await boolDao.findWithNullable(false);
expect(actual, equals(obj));
});

test('return nullables', () async {
await boolDao
.insertBoolC(BooleanClass(false, nullable: null, nonnullable: true));
await boolDao
.insertBoolC(BooleanClass(true, nullable: false, nonnullable: true));

expect(await boolDao.getNullables(), equals([null, false]));
});

test('return nonNullables', () async {
await boolDao
.insertBoolC(BooleanClass(false, nullable: null, nonnullable: true));
await boolDao
.insertBoolC(BooleanClass(true, nullable: false, nonnullable: false));

expect(await boolDao.getNonNullables(), equals([true, false]));
});
});
}

Expand Down Expand Up @@ -108,6 +126,12 @@ abstract class BoolDao {
@Query('SELECT * FROM BooleanClass where nonnullable = :val')
Future<BooleanClass> findWithNonNullable(bool val);

@Query('SELECT nonnullable FROM BooleanClass')
Future<List<bool>> getNonNullables();

@Query('SELECT nullable FROM BooleanClass')
Future<List<bool>> getNullables();

@Query('SELECT * FROM BooleanClass where nullable = :val')
Future<BooleanClass> findWithNullable(bool val);

Expand Down
Loading