Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Log email attachments #4108

Open
FelixMalfait opened this issue Feb 21, 2024 · 10 comments
Open

Log email attachments #4108

FelixMalfait opened this issue Feb 21, 2024 · 10 comments
Assignees
Labels
good for experienced contributor scope: backend Issues that are affecting the backend side only

Comments

@FelixMalfait
Copy link
Member

FelixMalfait commented Feb 21, 2024

Context

We've built an integration with gmail which allows workspace members to connect their accounts and import all emails. Those emails are then automatically attached to company/contacts.

The gmail API exposes attachments for every message but we ignore them.

The goal of this issue would be to import attachments in our database to then display them in the Files tab and in email threads.

Backend

The steps on the backend side should roughly look like this:

  1. STORAGE_DRIVER env variable is now used to store the default driver only
  2. Add new StorageDriverType for gmail
  3. Adapt fileService/fileStorageService to stream from Gmail's API instead of s3
  4. Add a new field storageDriver on Attachments
  5. Add new relation to messages on Attachments
  6. Adapt the gmail fetch service to create attachments along with company and people during import

This is just a general guideline, this a complex issue that will require investigation!

Frontend

  1. Because we use the existing Attachment object it should automatically appear in Files tabs (nothing to do?)

  2. We should also add a clipper icon on the threads list

Screenshot 2024-02-23 at 13 56 45
  1. We should show it on thread level (we display it as a field/fieldValue like for the properties of a task for example)
Screenshot 2024-02-23 at 13 58 24
  1. We should show it on message level
Screenshot 2024-02-23 at 13 59 33

Link to Figma

@FelixMalfait FelixMalfait added scope: backend Issues that are affecting the backend side only good for experienced contributor labels Feb 21, 2024
@tatethurston
Copy link
Contributor

@quest-bot embark

Copy link

quest-bot bot commented Mar 8, 2024

⚠️ There's no active Quest for this Issue.

Check the docs for more info.

@tatethurston
Copy link
Contributor

tatethurston commented Mar 8, 2024

@FelixMalfait do you all use any centralized access for shared secrets? Specifically here I needed AUTH_GOOGLE_CLIENT_ID and AUTH_GOOGLE_CLIENT_SECRET. I went through the process of spinning up a personal google integration, but I'm curious about this for future PRs.

@FelixMalfait
Copy link
Member Author

@tatethurston sorry for the delayed reply! We don't really have better process no, every just setup their own. I don't know if it would be possible to do otherwise 🤔

@tatethurston
Copy link
Contributor

No worries. I have limited capacity and don’t think I’ll be able to push this one through for a few weeks. Happy to let someone else take this on.

@varunKT001
Copy link

I would like to work on this issue.

@rostaklein
Copy link
Contributor

rostaklein commented Apr 2, 2024

Hey guys, i just started looking into this one. Please @FelixMalfait assign this issue to me. I was able to set up my own Google credentials and sync a few emails. Is there any preferred way to work with the email sync other than calling a manual command e.g. yarn nx command twenty-server workspace:gmail-partial-sync -w 20202020-1c25-4d02-bf25-6aeccf7ea419? 🤔 Any more tips/tricks on how to get emails visible in the UI and so (any reference PR)? 😊

@FelixMalfait
Copy link
Member Author

@varunKT001 thanks! Since this issue is a very complex one, you might want to start with smaller issues first. @rostaklein has some experience on the code base so I'm assigning it to him, but feel free to take another one!

cc @bosiraphael any tip to give to @rostaklein? thanks

@varunKT001
Copy link

@FelixMalfait sure no issues. I'll pick another issue 👍

@bosiraphael
Copy link
Contributor

bosiraphael commented Apr 3, 2024

Hello @rostaklein, thank you for taking this issue :) Here are some insights:

Currently, we're querying the raw email from gmail API using mailparser to parse the email content and headers.
The raw email contains the whole attachment but does not contain the gmail api attachment id.
Since we do not want to store the attachment but a way to get the attachment from the gmail api, we have two solutions:

Make a second query for files with attachments (Simpler)

  • Query the email in raw format
  • If the message has an attachment, make a second query to get the attachmentId
  • Save attachements after saving the messages and store gmailAttachmentId in the attachment table
  • We will be able to get the attachments using users.messages.attachments.get

This solution is simpler but we're wasting bandwidth because we're getting the whole attachment and doing nothing with it, and we also have to do a second query.

Change the query format to full instead of raw and modify the parsing logic (Better)

  • Change query format to full instead of raw in users.messages.get in fetch-messages-by-batches.service
  • Use adressParser instead of simpleParser to parse address objects (from, to, cc, bcc) and update the way we save messages in formatBatchResponseAsGmailMessage in fetch-messages-by-batches.service
  • Since we're not using mailParser simpleParser anymore, implement a function to parse the different message parts and get the whole message plain text
  • Save attachements information after saving the messages and store gmailAttachmentId in the attachment table
  • We will be able to get the attachments using users.messages.attachments.get

charlesBochet added a commit that referenced this issue May 20, 2024
first part of #4108
related PR #5081

---------

Co-authored-by: Charles Bochet <[email protected]>
srp-pawar added a commit to synapsenet-arena/lead360 that referenced this issue May 21, 2024
commit bb2d74f
Merge: b9c83a4 a981344
Author: Shubham Pawar <[email protected]>
Date:   Tue May 21 10:37:10 2024 +0530

    Merge branch 'main' of https://github.com/synapsenet-arena/lead360

commit a981344
Author: rostaklein <[email protected]>
Date:   Mon May 20 17:29:35 2024 +0200

    feat: fetch and parse full gmail message (twentyhq#5160)

    first part of twentyhq#4108
    related PR twentyhq#5081

    ---------

    Co-authored-by: Charles Bochet <[email protected]>

commit b5d3396
Author: bosiraphael <[email protected]>
Date:   Mon May 20 17:19:21 2024 +0200

    5477 - Introduce syncsubstatus in db to refactor gmail sync behavior (twentyhq#5479)

    Closes twentyhq#5477

commit 737fffe
Author: Indrakant D <[email protected]>
Date:   Mon May 20 20:29:01 2024 +0530

    Fix: danger font color values & acc. to design specs (twentyhq#5344)

    fixes: twentyhq#5325

    changes done (commits in order):

    1. **Fixed fontLight & fontDark 'danger' color as per design spec**:
    changed theme.font.color.danger to match the disabled color theme (for
    light and dark) as followed by the BorderDark and BorderLight. Use the
    updated colors for Buttons

    2. **Replace theme.font.color.danger with theme.color.red (5 changed
    files)**: Since `theme.font.color.danger` has now been updated to
    contain the disabled button color values, we use the `theme.color.red`
    color in all the places using `theme.font.color.danger` as it contains
    same value that was used to be of `theme.font.color.danger` before.

    3. **fixed hover color of StyledConfirmationButton in
    ConfirmationModal**: issue can be seen when going to /settings/workspace
    and trying to hover on delete the workspace button in dark mode. fixed
    with this commit.

    **Important Note**: The files
    `/twenty-front/src/modules/ui/theme/constants/FontLight.ts` and
    `/twenty-front/src/modules/ui/theme/constants/FontDark.ts` **are of no
    use** as theme for the entire 'twenty-front' and
    'twenty-chrome-extension' packages use the same files from '@/ui/theme'
    (twenty-ui package)

    dark mode :
    <img width="987" alt="Screenshot 2024-05-09 at 9 14 35 PM"
    src="https://github.com/twentyhq/twenty/assets/60315832/75fe3972-0e8a-41f6-90a1-09bfcd013e72">

    when disabled:
    <img width="1098" alt="Screenshot 2024-05-09 at 9 13 46 PM"
    src="https://github.com/twentyhq/twenty/assets/60315832/5caab8b5-47ba-43e5-90cd-a41a1f690ca0">

    on hover:
    <img width="1052" alt="Screenshot 2024-05-09 at 9 14 05 PM"
    src="https://github.com/twentyhq/twenty/assets/60315832/58de3df6-ed77-4aad-84fc-67b01154b493">

    <br>

    <br>

    light mode (when disabled):
    <img width="918" alt="Screenshot 2024-05-09 at 9 13 14 PM"
    src="https://github.com/twentyhq/twenty/assets/60315832/18228783-d6c7-44a6-9fce-00053bb35ef2">

    on hover:
    <img width="983" alt="Screenshot 2024-05-09 at 9 14 18 PM"
    src="https://github.com/twentyhq/twenty/assets/60315832/6df99f12-5767-4136-80c9-5d8883ac8e00">

    ---------

    Co-authored-by: Félix Malfait <[email protected]>

commit 4d479ee
Author: Thomas Trompette <[email protected]>
Date:   Mon May 20 16:37:35 2024 +0200

    Remove relations for remotes (twentyhq#5455)

    For remotes, we will only create the foreign key, without the relation
    metadata. Expected behavior will be:
    - possible to create an activity. But the remote object will not be
    displayed in the relations of the activity
    - the remote objects should not be available in the search for relations

    Also switched the number settings to an enum, since we now have to
    handle `BigInt` case.

    ---------

    Co-authored-by: Thomas Trompette <[email protected]>

commit b098027
Author: Weiko <[email protected]>
Date:   Mon May 20 15:53:13 2024 +0200

    Fix graphql prep query (twentyhq#5478)

commit 88f5eb6
Author: martmull <[email protected]>
Date:   Mon May 20 12:11:38 2024 +0200

    4689 multi workspace i should be able to accept an invite if im already logged in (twentyhq#5454)

    - split signInUp to separate Invitation from signInUp
    - update redirection logic
    - add a resolver for userWorkspace
    - add a mutation to add a user to a workspace
    - authorize /invite/hash while loggedIn
    - add a button to join a workspace

    ### Base functionnality

    https://github.com/twentyhq/twenty/assets/29927851/a1075a4e-a2af-4184-aa3e-e163711277a1

    ### Error handling

    https://github.com/twentyhq/twenty/assets/29927851/1bdd78ce-933a-4860-a87a-3f1f7bda389e

commit 1ceeb68
Author: ktang520 <[email protected]>
Date:   Mon May 20 02:31:39 2024 -0700

    Changed record chip functionality from onClick to anchor tag (twentyhq#5462)

    [twentyhq#4422](twentyhq#4422)

    Demo:

    https://github.com/twentyhq/twenty/assets/155670906/f8027ab2-c579-45f7-9f08-f4441a346ae7

    Within the demo, we show the various areas in which the Command/CTRL +
    Click functionality works. The table cells within the People and
    Companies tab open within both the current tab and new tab due to
    unchanged functionality within RecordTableCell. We did this to ensure we
    could get a PR within by the end of the week.

    In this commit, we ONLY edited EntityChip.tsx. We did this by:

    - Removing useNavigate() and handleLinkClick/onClick functionality

    - Wrapping InnerEntityChip in an anchor tag

    This allowed for Command/CTRL + Click functionality to work. Clickable
    left cells on tables, left side menu, and data model navigation
    files/areas DID NOT get updated.

    ---------

    Co-authored-by: Félix Malfait <[email protected]>

commit 8b5f79d
Author: Jérémy M <[email protected]>
Date:   Mon May 20 11:01:47 2024 +0200

    fix: multiple twenty orm issues & show an example of use (twentyhq#5439)

    This PR is fixing some issues and adding enhancement in TwentyORM:

    - [x] Composite fields in nested relations are not formatted properly
    - [x] Passing operators like `Any` in `where` condition is breaking the
    query
    - [x] Ability to auto load workspace-entities based on a regex path

    I've also introduced an example of use for `CalendarEventService`:

    https://github.com/twentyhq/twenty/pull/5439/files#diff-3a7dffc0dea57345d10e70c648e911f98fe237248bcea124dafa9c8deb1db748R15

commit 81e8f49
Author: H0onnn <[email protected]>
Date:   Mon May 20 00:26:29 2024 +0900

    Feat : Change title color of release page in dark mode (twentyhq#5467)

    ## Issue

    - close twentyhq#5459

    ## Work Detail

    Change title color of release page in dark mode.

    I worked using the useColorScheme and useSystemColorSheme hooks, but if
    there is a better way, please recommend it.

    ## Before
    <img width="606" alt="image"
    src="https://github.com/twentyhq/twenty/assets/116232939/f5c05360-f1d5-4701-b17d-e3e8a1db65fa">

    ## After
    <img width="565" alt="image"
    src="https://github.com/twentyhq/twenty/assets/116232939/5f9460d3-db62-461f-b7c2-659a4b687ba9">

    ---------

    Co-authored-by: Félix Malfait <[email protected]>

commit 66637a3
Author: Weiko <[email protected]>
Date:   Sat May 18 08:00:00 2024 +0200

    Add more details to mutation limit exception message and fix update many query (twentyhq#5460)

    ## Context
    Since we rely on PgGraphql to query the DB, we have to map its errors to
    more comprehensible errors before sending them back to the FE. This has
    already been done for unicity constraint and mutation maximum records
    but for the last one the message wasn't clear enough. This PR introduces
    a new pgGraphqlConfig param to the util to pass down the 'atMost' config
    that we are actually overwriting with an
    'MUTATION_MAXIMUM_RECORD_AFFECTED' env variable. See how atMost works in
    this doc (https://supabase.github.io/pg_graphql/api/#delete)

    Also adding the same message for the update since this mutation is also
    affected. Create is not though.

    Lastly, this PR introduces a fix on the updateMany. Since the current FE
    is not using updateMany, this was missed for a few weeks but a
    regression has been introduced when we started checking if the id is a
    valid UUID however for updateMany this was checking the data object
    instead of the filter object. Actually, the data object should never
    contain id because it wouldn't make sense to allow the update of the id
    and even more for multiple records since the id should be unique.

    ## Test
    locally with MUTATION_MAXIMUM_RECORD_AFFECTED=5

    <img width="1408" alt="Screenshot 2024-05-18 at 02 11 59"
    src="https://github.com/twentyhq/twenty/assets/1834158/06bf25ce-4a44-4851-8456-aed7689bb33e">
    <img width="1250" alt="Screenshot 2024-05-18 at 02 12 10"
    src="https://github.com/twentyhq/twenty/assets/1834158/06fc4329-147b-4bb4-9223-c3bce340a8d2">
    <img width="1222" alt="Screenshot 2024-05-18 at 02 12 36"
    src="https://github.com/twentyhq/twenty/assets/1834158/0674546e-73e2-4e5c-918f-9825f2ee5967">
    <img width="1228" alt="Screenshot 2024-05-18 at 02 13 01"
    src="https://github.com/twentyhq/twenty/assets/1834158/f50df435-1fd4-45df-a953-8fefa8f36e75">
    <img width="1174" alt="Screenshot 2024-05-18 at 02 13 09"
    src="https://github.com/twentyhq/twenty/assets/1834158/707b9300-2779-43df-8177-9658b8965b49">

    <img width="1393" alt="Screenshot 2024-05-18 at 02 19 11"
    src="https://github.com/twentyhq/twenty/assets/1834158/2cd167b6-1261-4914-a4db-36f792d810c0">

commit 0e525ca
Author: gitstart-app[bot] <57568882+gitstart-app[bot]@users.noreply.github.com>
Date:   Fri May 17 16:36:28 2024 +0200

    Implement <ScrollRestoration /> (twentyhq#5086)

    ### Description

    Implement &lt;ScrollRestoration /&gt;

    ### Refs

    [twentyhq#4357

    ### Demo

    https://github.com/twentyhq/twenty/assets/140154534/321242e1-4751-4204-8c86-e9b921c1733e

    Fixes twentyhq#4357

    ---------

    Co-authored-by: gitstart-twenty <[email protected]>
    Co-authored-by: Lucas Bordeau <[email protected]>
    Co-authored-by: v1b3m <[email protected]>
    Co-authored-by: RubensRafael <[email protected]>

commit 992602b
Author: Thaïs <[email protected]>
Date:   Fri May 17 16:05:31 2024 +0200

    fix: fix storybook build cache not being used by tests in CI (twentyhq#5451)

    TL;DR:
    - removed `--configuration={args.scope}` from `storybook:static:test`
    for the `storybook:static` part, as it was making `front-sb-test` jobs
    in CI not reuse the cache from the `front-sb-build` job and re-build
    storybook every time.
    - replaced it with a new `test` configuration which optimizes storybook
    build for tests and builds storybook 2x faster.

    ## Fix storybook:build cache usage in CI

    `storybook:static:test` executes two scripts in parallel:
    1. `storybook:static`, which depends on `storybook:build`
    1.a. it builds storybook first with `storybook:build`, the output
    directory is `storybook-static`.
    1.b. then it launches an `http-server`, using what has been built in
    `storybook-static`
    2. `storybook:test` to execute tests (needs the storybook http-server to
    be running)

    When passing `--configuration=pages` or `--configuration=modules` to
    `storybook:static` from step 1, those configurations are passed to the
    `storybook:build` script from step 1.a as well.

    But for Nx `storybook:build` and `storybook:build --configuration=pages`
    (or `modules`) are not the same command, therefore one does not reuse
    the cache of the other because they could output completely different
    things.

    As `front-sb-test` jobs are passing `--configuration={args.scope}` to
    `storybook:static`, the cache of the previously executed
    `storybook:build` (from `front-sb-build`) is not reused and therefore
    each job re-builds Storybook with its own scope, which increases CI
    time.

    ### Solution

    - Removed scope configurations from `storybook:static` and
    `storybook:build` scripts to avoid confusion.
    - `storybook:test` and `storybook:dev` can keep scope configurations as
    they can be useful and this doesn't impact storybook build cache in CI.

    ### Improve Storybook build time for testing

    Added the `test` configuration to `storybook:build` and
    `storybook:static` which makes Storybook build time 2x faster. It
    disables addons that slow down build time and are not used in tests.

commit 36e5411
Author: Thomas Trompette <[email protected]>
Date:   Fri May 17 10:38:17 2024 +0200

    Enable remotes with existing name (twentyhq#5433)

    - Check if a table with the same name already exists
    - If yes, add a number suffix, and check again

    Co-authored-by: Thomas Trompette <[email protected]>

commit 58f8c31
Author: Félix Malfait <[email protected]>
Date:   Fri May 17 09:10:59 2024 +0200

    Fixes typo in docs twentyhq#5076 (twentyhq#5450)

    Micro fix

    Fixes twentyhq#5076
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good for experienced contributor scope: backend Issues that are affecting the backend side only
Projects
Status: 🆕 New
Development

Successfully merging a pull request may close this issue.

5 participants