Integrate Spanner #1344
Draft
Integrate Spanner #1344
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This reverts commit 9713ba0.
|
Using multiple dbs with django spanner library will have some problems like googleapis/python-spanner-django#783 ... first of all, I check if using spanner is good for this project with raw SQLs ... |
- Renamed index on AdvancedSearchEntry from 'idx_entry_name' to 'AdvancedSearchEntryNameIndex'. - Modified AdvancedSearchAttributeValue table to change 'Value' from JSON to STRING(MAX) and added 'RawValue' as JSON. - Introduced 'Value_Tokens' as a hidden tokenized column for full-text search indexing on 'Value'. - Created a new search index 'AdvancedSearchAttributeValueValueIndex' on the tokenized 'Value_Tokens' column.
…ute handling - Streamlined entry processing in AdvancedSearchService by consolidating entry and attribute value insertion into a single batch operation. - Introduced a new class method `create_instance` in AdvancedSearchAttributeValue to encapsulate the logic for creating attribute value instances based on entity attributes and their values. - Updated the Spanner repository to accommodate changes in the AdvancedSearchAttributeValue model, including the addition of a `raw_value` field for storing JSON data. - Removed commented-out seed data insertion commands from the Spanner initialization script for clarity.
…bute handling and configuration - Changed the project ID setting in AdvancedSearchService to use AIRONE_SPANNER_PROJECT. - Enhanced attribute retrieval in AdvancedSearchService by querying all EntityAttr objects in a single query and using actual attribute names instead of placeholders. - Modified SpannerRepository to remove JSON loading for raw values, simplifying the data handling process. - Updated AdvancedSearchAPI to check for AIRONE_SPANNER_ENABLED instead of ENABLE_ESLESS_ADVANCED_SEARCH for conditional logic.
- Added 'name' field to AdvancedSearchAttribute model for better attribute identification. - Updated AdvancedSearchService to utilize the new 'name' field when organizing attributes by entry. - Modified SpannerRepository to include 'name' in SQL queries and results, improving data retrieval. - Altered Spanner schema to include 'Name' column in the AdvancedSearchAttribute table for consistency.
…dling - Added 'raw_value' to attributes in AdvancedSearchService for improved data representation. - Updated value assignment logic to prioritize 'raw_value' when available. - Introduced a helper function in SpannerRepository to parse raw values, ensuring proper handling of JSON objects and arrays. - Adjusted data retrieval in SpannerRepository to utilize the new parsing function for raw values.
…tes support - Added 'hint_attrs' parameter to AdvancedSearchService and SpannerRepository for improved search functionality. - Implemented filtering logic in SpannerRepository to handle various conditions based on hint attributes. - Updated attribute type conversion comments for clarity in AdvancedSearchService.
…ry counting and query conditions - Added a total count retrieval mechanism in AdvancedSearchService to return accurate entry counts. - Refactored SpannerRepository by introducing a new method to build common query conditions for both search and count operations. - Updated the search_entries method to utilize the new query conditions method, enhancing code clarity and maintainability. - Implemented a count_entries method in SpannerRepository to facilitate counting entries based on search criteria.
…a handling - Commented out MySQL indexing logic in AdvancedSearchService to accelerate indexing processes. - Implemented chunked processing for entries in AdvancedSearchService to optimize batch operations. - Updated SpannerRepository to use a unified SpannerOperation protocol for batch and mutation group operations, enhancing code clarity and maintainability. - Adjusted insert and delete methods in SpannerRepository to accommodate the new operation protocol.
…tes functionality - Introduced support for join attributes in AdvancedSearchService to allow complex queries involving related entities. - Updated SpannerRepository with a new method to retrieve joined entries based on specified join attributes. - Refactored attribute handling in AdvancedSearchService to improve data structure and readability. - Adjusted API view to accommodate the new join attributes parameter for advanced search queries.
- Refactored SpannerRepository query methods to use EXISTS clause for attribute filtering - Removed unnecessary JOINs in search and count queries to improve performance - Added new indexes on AdvancedSearchEntry, AdvancedSearchAttribute, and AdvancedSearchAttributeValue tables - Simplified query conditions generation for more efficient database querying
- Conditionally skip join attribute processing when Spanner is enabled - Preserve existing join attribute logic for non-Spanner environments - Improve code readability by adding a conditional block for join attribute processing
- Refactored join attribute retrieval method to use a single, more efficient SQL query - Simplified data processing and entry reconstruction logic - Improved performance by reducing the number of database calls - Maintained existing join attribute filtering and result generation behavior
- Update services and repository methods to use AdvancedSearchJoinAttrInfo serializer - Replace dict-based join attribute handling with strongly-typed attributes - Improve type safety and readability of advanced search join attribute processing - Maintain existing join attribute filtering and result generation behavior
- Correct total count calculation when join attributes are used with Spanner - Ensure ret_count reflects the actual number of returned values - Add a specific handling for Spanner-enabled environments with join attributes
- Add support for tracking referral relationships between entries in Spanner - Create new AdvancedSearchEntryReferral table to store referral connections - Implement method to convert origin entry IDs to Spanner entry IDs - Add a property graph to represent referral relationships - Enhance error handling for batch write operations - Collect and process referral relationships during entry synchronization
…rmation - Update entry referral data structure to include attribute ID - Modify Spanner schema to support attribute-specific referral tracking - Update SQL queries and repository methods to handle new referral structure - Improve referral tracking granularity in advanced search functionality
Improved the implementation of join attribute queries in SpannerRepository: - Simplified and optimized the query for fetching joined entries - Enhanced parameter handling for parent and child attributes - Improved query structure to handle joined attribute retrieval more efficiently - Removed unnecessary code and simplified logic in get_joined_entries method
- Modified search count query to use DISTINCT for accurate entry counting - Improved join attribute handling in count and search methods - Added EXISTS clause for parent attribute filtering - Refined query conditions to avoid duplicate attribute conditions
|
Experimentally I added a Spanner graph database The graph image is like: flowchart TD
A["Entry A"]
B["Entry B"]
C["Entry C"]
A -- "Referral (Attr: hoge)" --> B
A -- "Referral (Attr: fuga)" --> B
A -- "Referral (Attr: foo)" --> C
B -- "Referral (Attr: bar)" --> C
See |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Try Spanner to accelerate some features (advanced search, search chain, ...)