We can discuss roadmap in dev@griffin.incubator.apache.org

 

FeaturesApache Release
Accuracy-Batch

griffin-0.1.5-incubating

 

Accuracy-Streaming

griffin-0.1.6-incubating
Profilinggriffin-0.1.6-incubating

Uniqueness

griffin-0.2.0-incubating
Timelinessgriffin-0.2.0-incubating
Batch Job Schedulegriffin-0.2.0-incubating

Streaming Job Schedule

 
Completeness 
Consistency 
Validity 

 

 

 

 

  • No labels

7 Comments

  1. jianwen.fan

    质量监控方面，有如下需求：

    1、  波动检查

    2、  空值检查

    3、  枚举值检查

    4、  主键冲突

    5、  Missing value

    6、  ETL 数据延迟

    7、 平衡性检查

    8、基于历史趋势的异常检测

     

    结合开发的进度，帮忙给排一下期

  2. jianwen.fan

    Data quality monitoring, the following requirements:

    1.Period  check

    2.Null value check

    3.Enumeration value check

    4.Primary Key conflict

    5.Missing value

    6.ETL data delay

    7.Balance check

    8.Anomaly detection based on historical trend

     

    Combined with the progress of the development, to help schedule

     

     

  3. William Guo

    hi Fan,

    Could you elaborate more about item 1/7/8, adding more to describe the cases?

     

    Thanks,

    William

  4. 诸葛子房

    hi ,i see the project and find you use the framework of spring boot, in front of project ,i also use the spring boot,but i find the frame is not mature. I think springmvc is more better .

  5. William Guo

    could you specify what is the problem in current solution?

     

  6. 诸葛子房

    vip Data Quality Platform(DQP)
    1.datasources:
    (1)hive,postgresql,hive,oracle
    (2)api
    (3)redis
    (4)file(excle ect)
    2.measure
    we use python and sql ,it may be flexible;but it also has problem.it need the person who use the platform has high skills.
    3.project
    (1)hive:For simple rules, we adopt the adoption of direct adoption on hive
    (2)mysql:For some of the more complex business scenarios, we need to extract the data from the hive through the ETL to the MySQL and then perform it
    4.etl
    (1)database
    (2)api
    (3)redis
    (4)file
    5.authorization
    6.alert
    send email and message

  7. Soldier Shen

    William Guo  thanks for building such a great framework.  So do we have a rough plan on new feature/design?  I also see the "Griffin Improvement Proposals"