Add document transformation capability for indexing#123
Add document transformation capability for indexing#123tharropoulos wants to merge 9 commits intotypesense:masterfrom
Conversation
|
|
||
| const url = `https://${region}-${projectId}.cloudfunctions.net/${transformFunctionName}`; | ||
|
|
||
| const response = await fetch(url, { |
There was a problem hiding this comment.
@tharropoulos Can we add automatic retries here with exponential backoff?
| example: transformDoc | ||
| default: "" | ||
| required: false | ||
| - param: TRANSFORM_FUNCTION_PROJECT_ID |
There was a problem hiding this comment.
@tharropoulos Given the number of new parameters this will add to the UI, what do you think about adding a constraint so that the function has to be in the same project and region as the one the extension is installed in?
I know it will limit some functionality, but it will also help keep the config simple.
Unless, is there a way to move this under an advanced collapsible section in the installation UI?
- add `transformDocument` function to call external transformation functions - implement error handling for failed transformations with fallback to original doc - add comprehensive tests for the transformation functionality - update package.json with new test command
- update indexOnWrite to conditionally use transformation function - check for transformFunctionName in config before transforming - apply transformation before converting to typesense document
- add TRANSFORM_FUNCTION_NAME parameter for specifying transform function - add TRANSFORM_FUNCTION_PROJECT_ID for cross-project function support - add TRANSFORM_FUNCTION_REGION for region specification - add TRANSFORM_FUNCTION_SECRET for authorization to protected functions
- Remove `TRANSFORM_FUNCTION_PROJECT_ID` and `TRANSFORM_FUNCTION_REGION` parameters from `extension.yaml` - Update `config.js` to use `GCLOUD_PROJECT` and `LOCATION` environment
tharropoulos
force-pushed
the
transform-func
branch
from
4579313 to
f5c6b8e
Compare
|
|
||
| const url = `https://${region}-${projectId}.cloudfunctions.net/${transformFunctionName}`; | ||
|
|
||
| const response = await fetch(url, { |
| transformFunctionName: process.env.TRANSFORM_FUNCTION_NAME, | ||
| transformFunctionSecret: process.env.TRANSFORM_FUNCTION_SECRET, | ||
| transformFunctionProjectId: process.env.GCLOUD_PROJECT, | ||
| transformFunctionRegion: process.env.LOCATION, |
There was a problem hiding this comment.
Are the above two lines still needed?
There was a problem hiding this comment.
We have to call out to eh region-project id. Both of those are coming from env variables. The gcloud project is automatic and the location is being set from extension.yaml.
2 participants
TLDR
New options to transform Firestore documents before indexing to Typesense.
Change Summary
Added Configuration:
extension.yaml:TRANSFORM_FUNCTION_NAME: Name of Cloud Function for document transformationTRANSFORM_FUNCTION_PROJECT_ID: Project where transform function is deployedTRANSFORM_FUNCTION_REGION: Region of transform functionTRANSFORM_FUNCTION_SECRET: Auth secret for transform functionAdded Functionality:
In
utils.js:transformDocument(): Calls external transform function with error handlingIn
indexOnWrite.js:Added Tests:
New file
test/utilsTransform.spec.js:In
package.json:test:utilsPR Checklist