spring-cloud-gateway基于spring-cloud-starter-loadbalancer的灰度发布

2021-07-10 spring Spring-Cloud, gateway, nacos, spring, springboot, springcloud, 灰度, 网关 0 0 评论

前言

之前写过 spring-cloud-gateway系列文章, 今天再来补充一篇关于灰度发布的。传送门:

今天的文章主题会按照如下三步曲进行记录:

基于nacos的基础项目搭建
动态路由
灰度方案

之前的文章吃了有些没有版本号和文档上传不完整的亏, 导致很多网友咨询的时候我也不记得细节, 这里我们会贴出完整配置及代码。

此处的灰度方案只是简单的网关灰度演示, 开发中服务间的调用也存在灰度, 后续再分解。

基础环境搭建

很久没有搭建 SpringCloud 项目了, 首先搭建一个基础示例, 供代码调试。

首先, 快速搭建nacos-server, 这里基于官方文档压缩包安装。安装完成后访问 http://127.0.0.1:8848/nacos 即可看到 nacos 控制面板
项目创建。我们计划创建2个服务, 一个服务提供者 provider，一个网关 gateway。项目采用聚合项目的形式。

根目录pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>cn.idea360</groupId>
    <artifactId>spring-cloud-demo</artifactId>
    <version>0.0.1</version>
    <packaging>pom</packaging>

    <properties>
        <maven.compiler.source>11</maven.compiler.source>
        <maven.compiler.target>11</maven.compiler.target>
        <spring-boot.version>2.3.7.RELEASE</spring-boot.version>
        <spring-cloud-alibaba.version>2.2.2.RELEASE</spring-cloud-alibaba.version>
        <spring-cloud.version>Hoxton.SR9</spring-cloud.version>
    </properties>

    <modules>
        <module>provider</module>
        <module>gateway</module>
    </modules>

    <dependencies>
        <dependency>
            <groupId>com.alibaba.cloud</groupId>
            <artifactId>spring-cloud-starter-alibaba-nacos-config</artifactId>
        </dependency>
        <dependency>
            <groupId>com.alibaba.cloud</groupId>
            <artifactId>spring-cloud-starter-alibaba-nacos-discovery</artifactId>
        </dependency>
    </dependencies>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.cloud</groupId>
                <artifactId>spring-cloud-dependencies</artifactId>
                <version>${spring-cloud.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
            <dependency>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-dependencies</artifactId>
                <version>${spring-boot.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
            <dependency>
                <groupId>com.alibaba.cloud</groupId>
                <artifactId>spring-cloud-alibaba-dependencies</artifactId>
                <version>${spring-cloud-alibaba.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>
</project>

服务提供者provider

这里我们计划引入 nacos, 所以先创建一个nacos配置文件 dataId 为 provider.properties, 这里用默认的命名空间 public, 默认分组 DEFAULT_GROUP

version=2

provider的pom文件如下:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd
http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <artifactId>provider</artifactId>

    <parent>
        <groupId>cn.idea360</groupId>
        <artifactId>spring-cloud-demo</artifactId>
        <version>0.0.1</version>
    </parent>
    <name>provider</name>
    <description>Demo project for Spring Boot</description>

    <properties>
        <java.version>1.8</java.version>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
        <spring-boot.version>2.3.7.RELEASE</spring-boot.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
            <exclusions>
                <exclusion>
                    <groupId>org.junit.vintage</groupId>
                    <artifactId>junit-vintage-engine</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.8.1</version>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                    <encoding>UTF-8</encoding>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <version>2.3.7.RELEASE</version>
                <configuration>
                    <mainClass>cn.idea360.provider.ProviderApplication</mainClass>
                </configuration>
                <executions>
                    <execution>
                        <id>repackage</id>
                        <goals>
                            <goal>repackage</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

</project>

application.properties配置:

# 应用名称
spring.application.name=provider
# 应用服务 WEB 访问端口
server.port=9001

spring.cloud.nacos.config.server-addr=127.0.0.1:8848
spring.cloud.nacos.discovery.server-addr=127.0.0.1:8848

在启动类, 我们增加了服务发现注解 @EnableDiscoveryClient

package cn.idea360.provider;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.client.discovery.EnableDiscoveryClient;

@EnableDiscoveryClient
@SpringBootApplication
public class ProviderApplication {

    public static void main(String[] args) {
        SpringApplication.run(ProviderApplication.class, args);
    }

}

controller逻辑很简单, 只是简单返回版本号

package cn.idea360.provider.controller;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.cloud.context.config.annotation.RefreshScope;
import org.springframework.core.env.Environment;
import org.springframework.web.bind.annotation.*;

/**
 * @author cuishiying
 * @date 2021-01-22
 */
@RefreshScope
@RestController
@RequestMapping("/test")
public class TestController {

    @Autowired
    private Environment env;

    @Value("${version:0}")
    private String version;
    /**
     * http://localhost:9001/test/port
     * @return
     */
    @GetMapping("/port")
    public Object port() {
        return String.format("port=%s, version=%s", env.getProperty("local.server.port"), version);
    }
}

网关gateway

gateway服务的pom配置如下:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <artifactId>gateway</artifactId>
    <parent>
        <groupId>cn.idea360</groupId>
        <artifactId>spring-cloud-demo</artifactId>
        <version>0.0.1</version>
    </parent>
    <name>gateway</name>
    <description>Demo project for Spring Boot</description>

    <properties>
        <java.version>1.8</java.version>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
        <spring-boot.version>2.3.7.RELEASE</spring-boot.version>
        <spring-cloud-alibaba.version>2.2.2.RELEASE</spring-cloud-alibaba.version>
        <spring-cloud.version>Hoxton.SR9</spring-cloud.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-gateway</artifactId>
        </dependency>

        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <optional>true</optional>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
            <exclusions>
                <exclusion>
                    <groupId>org.junit.vintage</groupId>
                    <artifactId>junit-vintage-engine</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.8.1</version>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                    <encoding>UTF-8</encoding>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <version>2.3.7.RELEASE</version>
                <configuration>
                    <mainClass>cn.idea360.gateway.GatewayApplication</mainClass>
                </configuration>
                <executions>
                    <execution>
                        <id>repackage</id>
                        <goals>
                            <goal>repackage</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

</project>

网关同样做服务发现

package cn.idea360.gateway;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.client.discovery.EnableDiscoveryClient;

@EnableDiscoveryClient
@SpringBootApplication
public class GatewayApplication {

    public static void main(String[] args) {
        SpringApplication.run(GatewayApplication.class, args);
    }

}

网关核心配置, 这里我们先采用静态路由的方式测试下项目是否有问题

application.yml

# 应用服务 WEB 访问端口
server:
  port: 9000
# 应用名称
spring:
  application:
    name: gateway
  cloud:
    nacos:
      config:
        server-addr: 127.0.0.1:8848
      discovery:
        server-addr: 127.0.0.1:8848
    gateway: 
      routes: # http://127.0.0.1:9000/actuator/gateway/routes
        - id: provider  # 路由 ID，保持唯一
          uri: lb://provider # uri指目标服务地址，lb代表从注册中心获取服务
          predicates:
            - Path=/provider/**  # http://127.0.0.1:9000/provider/port 会转发到 http://localhost:9001/provider/port, 和预期不符合, 需要StripPrefix来处理
          filters:
            - StripPrefix=1 # StripPrefix=1就代表截取路径的个数为1, 这样请求 http://127.0.0.1:9000/provider/test/port 会转发到 http://localhost:9001/test/port

接下来就是激动人心的时刻了, 先测试下服务是否ok.

➜  blog git:(master) ✗ curl http://localhost:9001/test/port
port=9001, version=2%
➜  blog git:(master) ✗ curl http://127.0.0.1:9000/provider/test/port
port=9001, version=2%

通过结果可见, 网关的基本配置已经生效了。

基于nacos动态路由

动态路由的实现有2种方式, 一个就是像之前一样改写 RouteDefinitionRepository, 一个就是基于 nacos 的监听器给 RouteDefinitionRepository 动态更新值。实现逻辑大同小异。

基于nacos监听器实现动态路由

配置nacos配置文件 gateway-router.json, 为了避免项目 application.yml 中配置文件影响, 先注释掉 gateway 相关的配置

[{
    "id": "provider",
    "predicates": [{
        "name": "Path",
        "args": {
            "_genkey_0": "/provider/**"
        }
    }],
    "filters": [{
        "name": "StripPrefix",
        "args": {
            "_genkey_0": "1"
        }
    }],
    "uri": "lb://provider",
    "order": 0
}]

基于 nacos 监听器实现路由刷新

路由配置

@Data
@ConfigurationProperties(prefix = "idea360.framework.gateway.route")
public class DynamicRouteProperties {

    /**
     * 动态路由配置分组
     */
    private String group;

    /**
     * 动态路由配置文件. eg: gateway-router.json
     */
    private String dataId;

具体实现

package cn.idea360.gateway.config;

import com.alibaba.nacos.api.NacosFactory;
import com.alibaba.nacos.api.config.ConfigService;
import com.alibaba.nacos.api.config.listener.Listener;
import com.alibaba.nacos.api.exception.NacosException;
import cn.idea360.gateway.route.properties.DynamicRouteProperties;
import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.ObjectMapper;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.InitializingBean;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.cloud.gateway.event.RefreshRoutesEvent;
import org.springframework.cloud.gateway.route.RouteDefinition;
import org.springframework.cloud.gateway.route.RouteDefinitionWriter;
import org.springframework.context.ApplicationEventPublisher;
import org.springframework.context.ApplicationEventPublisherAware;
import reactor.core.publisher.Mono;

import java.util.ArrayList;
import java.util.List;
import java.util.Properties;
import java.util.concurrent.Executor;

/**
 * @author cuishiying
 * @date 2021-01-21
 */
@Slf4j
public class NacosGatewayDynamicRoute implements ApplicationEventPublisherAware, InitializingBean {

    private ApplicationEventPublisher applicationEventPublisher;

    private ConfigService configService;

    private static final String SERVER_ADDR = "serverAddr";

    private static final String NAMESPACE = "namespace";

    @Value("${spring.cloud.nacos.config.server-addr}")
    private String serverAddr;

    @Value("${spring.cloud.nacos.config.namespace}")
    private String namespace;

    private final DynamicRouteProperties dynamicRouteProperties;

    private final RouteDefinitionWriter routeDefinitionWriter;

    private static final List<String> routeList = new ArrayList<>();

    private final ObjectMapper objectMapper = new ObjectMapper();

    public static final long DEFAULT_TIMEOUT = 30000;

    public NacosGatewayDynamicRoute(DynamicRouteProperties dynamicRouteProperties,
            RouteDefinitionWriter routeDefinitionWriter) {
        this.dynamicRouteProperties = dynamicRouteProperties;
        this.routeDefinitionWriter = routeDefinitionWriter;
    }

    public void initNacosConfig() {
        log.info("动态路由初始化...");
        try {
            Properties properties = new Properties();
            properties.setProperty(SERVER_ADDR, serverAddr);
            properties.setProperty(NAMESPACE, namespace);
            configService = NacosFactory.createConfigService(properties);
            if (configService == null) {
                log.warn("初始化配置服务失败...");
                return;
            }
            String configInfo = configService.getConfig(dynamicRouteProperties.getDataId(),
                    dynamicRouteProperties.getGroup(), DEFAULT_TIMEOUT);
            log.info("获取网关当前路由配置:\r\n{}", configInfo);
            updateRouteConfig(configInfo);
        }
        catch (Exception e) {
            log.error("初始化网关路由时发生错误", e);
        }
        dynamicRouteByNacosListener();
    }

    public void dynamicRouteByNacosListener() {
        try {
            configService.addListener(dynamicRouteProperties.getDataId(), dynamicRouteProperties.getGroup(),
                    new Listener() {

                        public void receiveConfigInfo(String configInfo) {
                            updateRouteConfig(configInfo);
                        }

                        public Executor getExecutor() {
                            return null;
                        }
                    });
        }
        catch (NacosException e) {
            log.error("动态更新网关路由配置错误", e);
        }
    }

    private void updateRouteConfig(String configInfo) {
        log.info("自动更新配置...\r\n{}", configInfo);
        clearRoute();
        try {
            List<RouteDefinition> gatewayRouteDefinitions = objectMapper.readValue(configInfo, new TypeReference<>() {
            });
            for (RouteDefinition routeDefinition : gatewayRouteDefinitions) {
                addRoute(routeDefinition);
            }
            publish();
        }
        catch (Exception e) {
            e.printStackTrace();
        }
    }

    private void clearRoute() {
        for (String id : routeList) {
            this.routeDefinitionWriter.delete(Mono.just(id)).subscribe();
        }
        routeList.clear();
    }

    private void addRoute(RouteDefinition definition) {
        try {
            routeDefinitionWriter.save(Mono.just(definition)).subscribe();
            routeList.add(definition.getId());
        }
        catch (Exception e) {
            e.printStackTrace();
        }
    }

    private void publish() {
        this.applicationEventPublisher.publishEvent(new RefreshRoutesEvent(this.routeDefinitionWriter));
    }

    @Override
    public void setApplicationEventPublisher(ApplicationEventPublisher applicationEventPublisher) {
        this.applicationEventPublisher = applicationEventPublisher;
    }

    @Override
    public void afterPropertiesSet() throws Exception {
        initNacosConfig();
    }

}

SPI注入

@Configuration
@EnableConfigurationProperties(DynamicRouteProperties.class)
public class DynamicRouteAutoConfiguration {

    @Bean
    @ConditionalOnBean({ RouteDefinitionWriter.class })
    public NacosGatewayDynamicRoute nacosGatewayDynamicRoute(DynamicRouteProperties dynamicRouteProperties,
            RouteDefinitionWriter routeDefinitionWriter) {
        return new NacosGatewayDynamicRoute(dynamicRouteProperties, routeDefinitionWriter);
    }

}

重启网关, 请求 http://127.0.0.1:9000/actuator/gateway/routes 可以发现配置已生效。

[
    {
        "predicate": "Paths: [/provider/**], match trailing slash: true",
        "route_id": "provider",
        "filters": [
            "[[StripPrefix parts = 1], order = 1]"
        ],
        "uri": "lb://provider",
        "order": 0
    }
]

路由测试结果如下:

1 2	➜ blog git:(master) ✗ curl http://127.0.0.1:9000/provider/test/port port=9001, version=2%

基于RouteDefinitionRepository实现动态路由

Spring Cloud Gateway 中加载路由信息分别由以下几个类负责

1、PropertiesRouteDefinitionLocator：从配置文件中读取路由信息(如YML、Properties等)
2、RouteDefinitionRepository：从存储器中读取路由信息(如内存、配置中心、Redis、MySQL等)
3、DiscoveryClientRouteDefinitionLocator：从注册中心中读取路由信息(如Nacos、Eurka、Zookeeper等)

package cn.idea360.gateway.router;

import com.alibaba.nacos.api.NacosFactory;
import com.alibaba.nacos.api.config.ConfigService;
import com.alibaba.nacos.api.config.listener.Listener;
import com.alibaba.nacos.api.exception.NacosException;
import cn.idea360.gateway.route.properties.DynamicRouteProperties;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.google.common.collect.Lists;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.cloud.gateway.event.RefreshRoutesEvent;
import org.springframework.cloud.gateway.route.RouteDefinition;
import org.springframework.cloud.gateway.route.RouteDefinitionRepository;
import org.springframework.context.ApplicationEventPublisher;
import org.springframework.context.ApplicationEventPublisherAware;
import reactor.core.publisher.Flux;
import reactor.core.publisher.Mono;

import java.util.List;
import java.util.Properties;
import java.util.concurrent.Executor;

/**
 * @author cuishiying
 * @date 2021-01-17
 */
@Slf4j
public class NacosRouteDefinitionRepository implements RouteDefinitionRepository, ApplicationEventPublisherAware {

    private ApplicationEventPublisher applicationEventPublisher;

    private final ObjectMapper objectMapper = new ObjectMapper();

    private ConfigService configService;

    private static final String SERVER_ADDR = "serverAddr";

    private static final String NAMESPACE = "namespace";

    @Value("${spring.cloud.nacos.config.server-addr}")
    private String serverAddr;

    @Value("${spring.cloud.nacos.config.namespace}")
    private String namespace;

    private final DynamicRouteProperties dynamicRouteProperties;

    public static final long DEFAULT_TIMEOUT = 30000;

    public NacosRouteDefinitionRepository(DynamicRouteProperties dynamicRouteProperties) {
        this.dynamicRouteProperties = dynamicRouteProperties;
        initNacosConfig();
    }

    public void initNacosConfig() {
        log.info("动态路由初始化...");
        try {
            Properties properties = new Properties();
            properties.setProperty(SERVER_ADDR, serverAddr);
            properties.setProperty(NAMESPACE, namespace);
            configService = NacosFactory.createConfigService(properties);
            if (configService == null) {
                log.warn("初始化配置服务失败...");
                return;
            }
        }
        catch (Exception e) {
            log.error("初始化网关路由时发生错误", e);
        }
        dynamicRouteByNacosListener();
    }

    public void dynamicRouteByNacosListener() {
        try {
            configService.addListener(dynamicRouteProperties.getDataId(), dynamicRouteProperties.getGroup(),
                    new Listener() {

                        public void receiveConfigInfo(String configInfo) {
                            log.info("自动更新配置...\r\n{}", configInfo);
                            applicationEventPublisher.publishEvent(new RefreshRoutesEvent(this));
                        }

                        public Executor getExecutor() {
                            return null;
                        }
                    });
        }
        catch (NacosException e) {
            log.error("动态更新网关路由配置错误", e);
        }
    }

    @Override
    public Flux<RouteDefinition> getRouteDefinitions() {
        try {
            String configInfo = configService.getConfig(dynamicRouteProperties.getDataId(),
                    dynamicRouteProperties.getGroup(), DEFAULT_TIMEOUT);
            log.info("获取网关当前路由配置:\r\n{}", configInfo);
            List<RouteDefinition> gatewayRouteDefinitions = objectMapper.readValue(configInfo, new TypeReference<>() {
            });
            return Flux.fromIterable(gatewayRouteDefinitions);
        }
        catch (NacosException | JsonProcessingException e) {
            log.error("网关配置获取失败", e);
        }
        return Flux.fromIterable(Lists.newArrayList());
    }

    @Override
    public Mono<Void> save(Mono<RouteDefinition> route) {
        return null;
    }

    @Override
    public Mono<Void> delete(Mono<String> routeId) {
        return null;
    }

    @Override
    public void setApplicationEventPublisher(ApplicationEventPublisher applicationEventPublisher) {
        this.applicationEventPublisher = applicationEventPublisher;
    }

}

@Component测试结果同上, 但是SPI注入失败未解决。

基于SpringCloud Gateway + nacos 灰度路由

首先需要明白灰度的场景, 因为有不同版本的服务需要共存, 所以新的节点升级的时候必然代码及配置会存在差别, 所以我们根据这种差别来判断服务版本是新版本还是线上稳定版本。这里我们用 prod 和 gray 来标识2个版本。

实现的整体思路：

编写带版本号的灰度路由(负载均衡策略)
编写自定义filter
nacos服务配置需要灰度发布的服务的元数据信息以及权重(在服务jar中配置)

注意, 应该先修改nacos配置实现动态路由, 然后再升级灰度节点. 本案例只是简单示例灰度原理。

网关配置

首先排除掉默认的ribbon

<dependency>
    <groupId>com.alibaba.cloud</groupId>
    <artifactId>spring-cloud-starter-alibaba-nacos-discovery</artifactId>
    <exclusions>
        <exclusion>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-netflix-ribbon</artifactId>
        </exclusion>
    </exclusions>
</dependency>

引入官方新的负载均衡包

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-loadbalancer</artifactId>
</dependency>

自定义负载均衡策略

package cn.idea360.gateway.gray;

import cn.idea360.gateway.gray.rule.GrayRuleManage;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.ObjectProvider;
import org.springframework.cloud.client.ServiceInstance;
import org.springframework.cloud.client.loadbalancer.reactive.DefaultResponse;
import org.springframework.cloud.client.loadbalancer.reactive.EmptyResponse;
import org.springframework.cloud.client.loadbalancer.reactive.Request;
import org.springframework.cloud.client.loadbalancer.reactive.Response;
import org.springframework.cloud.loadbalancer.core.NoopServiceInstanceListSupplier;
import org.springframework.cloud.loadbalancer.core.ReactorServiceInstanceLoadBalancer;
import org.springframework.cloud.loadbalancer.core.ServiceInstanceListSupplier;
import reactor.core.publisher.Mono;

import java.util.List;
import java.util.Random;
import java.util.concurrent.atomic.AtomicInteger;

/**
 * 基于nacos的灰度负载均衡器
 *
 * @author cuishiying
 * @date 2021-01-18
 */
@Slf4j
public class NacosGrayLoadBalancer implements ReactorServiceInstanceLoadBalancer {

    private final ObjectProvider<ServiceInstanceListSupplier> serviceInstanceListSupplierProvider;

    /**
     * 服务名
     */
    private final String serviceId;

    private final GrayRuleManage grayRuleManage;

    private final AtomicInteger position;

    public NacosGrayLoadBalancer(ObjectProvider<ServiceInstanceListSupplier> serviceInstanceListSupplierProvider,
            String serviceId, GrayRuleManage grayRuleManage) {
        this(serviceInstanceListSupplierProvider, serviceId, new Random().nextInt(1000), grayRuleManage);
    }

    public NacosGrayLoadBalancer(ObjectProvider<ServiceInstanceListSupplier> serviceInstanceListSupplierProvider,
            String serviceId, int seedPosition, GrayRuleManage grayRuleManage) {
        this.serviceInstanceListSupplierProvider = serviceInstanceListSupplierProvider;
        this.serviceId = serviceId;
        this.position = new AtomicInteger(seedPosition);
        this.grayRuleManage = grayRuleManage;
    }

    @SuppressWarnings("deprecation")
    @Override
    public Mono<Response<ServiceInstance>> choose(Request request) {
        ServiceInstanceListSupplier supplier = serviceInstanceListSupplierProvider
                .getIfAvailable(NoopServiceInstanceListSupplier::new);
        return supplier.get().next().map(instances -> getInstanceResponse(instances, request));
    }

    @SuppressWarnings("deprecation")
    private Response<ServiceInstance> getInstanceResponse(List<ServiceInstance> instances, Request request) {
        if (instances.isEmpty()) {
            log.warn("No servers available for service: {}", this.serviceId);
            return new EmptyResponse();
        }
        return grayRuleManage.choose(request, serviceId, instances);
    }

    /**
     * 轮询
     */
    @SuppressWarnings("deprecation")
    private Response<ServiceInstance> processRoundRobinInstanceResponse(List<ServiceInstance> instances) {
        int pos = Math.abs(this.position.incrementAndGet());
        ServiceInstance instance = instances.get(pos % instances.size());
        return new DefaultResponse(instance);
    }

}

灰度策略管理器

package cn.idea360.gateway.gray.rule;

import cn.idea360.gateway.gray.properties.GrayProperties;
import cn.idea360.gateway.gray.properties.GrayProperties.Gray;
import cn.idea360.gateway.gray.rule.impl.CompanyIdGrayRule;
import lombok.extern.slf4j.Slf4j;
import org.springframework.cloud.client.ServiceInstance;
import org.springframework.cloud.client.loadbalancer.reactive.Request;
import org.springframework.cloud.client.loadbalancer.reactive.Response;
import org.springframework.util.Assert;
import org.springframework.util.StringUtils;

import java.util.*;
import java.util.concurrent.ConcurrentHashMap;

/**
 * @author cuishiying
 * @date 2021-01-20
 */
@Slf4j
public class GrayRuleManage {

    /**
     * 内置灰度策略集合
     */
    private final Map<String, GrayRule> ruleMap = new ConcurrentHashMap<>();

    /**
     * 内置策略名
     */
    private final Set<String> ruleNames = new HashSet<>();

    /**
     * 网关灰度配置
     */
    private final GrayProperties grayProperties;

    public GrayRuleManage(GrayProperties grayProperties) {
        this.grayProperties = grayProperties;
        registGrayRules();
    }

    /**
     * 注册内置灰度策略
     */
    private void registGrayRules() {
        for (DefaultRule rule : DefaultRule.values()) {
            GrayRule grayRule = rule.newInstance();
            log.info("注册内置灰度策略 [{}]", grayRule.ruleName());
            ruleMap.put(grayRule.ruleName(), grayRule);
            ruleNames.add(grayRule.ruleName());
        }
    }

    /**
     * 新增灰度负载策略. 可以覆盖内置策略
     * @param grayRule 灰度策略
     */
    public void addGrayRule(GrayRule grayRule) {
        Assert.notNull(grayRule, "rule must not be null");
        Assert.state(!StringUtils.isEmpty(grayRule.ruleName()), "ruleName must not be empty");
        ruleMap.put(grayRule.ruleName(), grayRule);
        ruleNames.add(grayRule.ruleName());
    }

    /**
     * 灰度策略
     * @param request http请求
     * @param serviceId 服务名
     * @param instances 所有服务实例
     * @return 命中的实例
     */
    @SuppressWarnings("deprecation")
    public Response<ServiceInstance> choose(Request request, String serviceId, List<ServiceInstance> instances) {
        Assert.notNull(grayProperties.getGray(), "请检查'easyliao.framework.gateway.gray.**'配置是否正确");
        Assert.notNull(grayProperties.getGray().get(serviceId), String.format("[%s]服务未做灰度配置", serviceId));

        Gray gray = grayProperties.getGray().get(serviceId);
        if (Objects.isNull(gray.getGrayRule())) {
            gray.setGrayRule(CompanyIdGrayRule.RULE_NAME);
        }
        if (!ruleNames.contains(gray.getGrayRule())) {
            gray.setGrayRule(CompanyIdGrayRule.RULE_NAME);
        }
        log.info("服务[{}]配置的灰度策略为[{}]", serviceId, gray.getGrayRule());
        GrayRule grayRule = ruleMap.get(gray.getGrayRule());
        return grayRule.choose(request, serviceId, instances, grayProperties);
    }

}

灰度策略接口定义

/**
 * 灰度策略
 *
 * @author cuishiying
 * @date 2021-01-19
 */
public interface GrayRule {

    /**
     * @return 灰度策略名
     */
    String ruleName();

    /**
     * 灰度策略
     * @param request 请求
     * @param instances 所有服务实例
     * @param grayProperties 灰度配置
     * @return 命中的实例
     */
    @SuppressWarnings("deprecation")
    Response<ServiceInstance> choose(Request request, String serviceId, List<ServiceInstance> instances,
            GrayProperties grayProperties);

}

内置灰度策略注入

/**
 * @author cuishiying
 * @date 2021-01-19
 */
public enum DefaultRule {

    VERSION(CompanyIdGrayRule.class),;

    private final Class<? extends GrayRule> grayRuleClass;

    DefaultRule(Class<? extends GrayRule> grayRuleClass) {
        this.grayRuleClass = grayRuleClass;
    }

    public GrayRule newInstance() {
        return ClassUtils.newInstance(this.grayRuleClass);
    }

}

自定义过滤器加载负载均衡策略

package cn.idea360.gateway.gray;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.springframework.cloud.client.ServiceInstance;
import org.springframework.cloud.client.loadbalancer.LoadBalancerUriTools;
import org.springframework.cloud.client.loadbalancer.reactive.DefaultRequest;
import org.springframework.cloud.client.loadbalancer.reactive.Request;
import org.springframework.cloud.client.loadbalancer.reactive.Response;
import org.springframework.cloud.gateway.config.LoadBalancerProperties;
import org.springframework.cloud.gateway.filter.GatewayFilterChain;
import org.springframework.cloud.gateway.filter.GlobalFilter;
import org.springframework.cloud.gateway.filter.ReactiveLoadBalancerClientFilter;
import org.springframework.cloud.gateway.filter.RouteToRequestUrlFilter;
import org.springframework.cloud.gateway.support.DelegatingServiceInstance;
import org.springframework.cloud.gateway.support.NotFoundException;
import org.springframework.cloud.loadbalancer.core.ReactorLoadBalancer;
import org.springframework.cloud.loadbalancer.core.ReactorServiceInstanceLoadBalancer;
import org.springframework.cloud.loadbalancer.support.LoadBalancerClientFactory;
import org.springframework.core.Ordered;
import org.springframework.http.HttpHeaders;
import org.springframework.web.server.ServerWebExchange;
import reactor.core.publisher.Mono;

import java.net.URI;

import static org.springframework.cloud.gateway.support.ServerWebExchangeUtils.*;

/**
 * 修改自源码 {@link ReactiveLoadBalancerClientFilter}. </br>
 * 重写了{@link GrayReactiveLoadBalancerClientFilter#createRequest(org.springframework.web.server.ServerWebExchange)}方法
 * </br>
 * 解决负载均衡器中无http请求的问题
 *
 * 测试中会首先执行
 * {@link ReactiveLoadBalancerClientFilter#filter(org.springframework.web.server.ServerWebExchange, org.springframework.cloud.gateway.filter.GatewayFilterChain)}
 * 所以结果会不符合预期, 会将lb:// 转换为 http://
 *
 * {@link RouteToRequestUrlFilter#filter(org.springframework.web.server.ServerWebExchange, org.springframework.cloud.gateway.filter.GatewayFilterChain)}
 * 转发之后的地址，写到了GATEWAY_REQUEST_URL_ATTR这个参数里
 *
 * @author cuishiying
 * @date 2021-01-18
 */
public class GrayReactiveLoadBalancerClientFilter implements GlobalFilter, Ordered {

    private static final Log log = LogFactory.getLog(ReactiveLoadBalancerClientFilter.class);

    private static final int LOAD_BALANCER_CLIENT_FILTER_ORDER = 10150;

    private final LoadBalancerClientFactory clientFactory;

    private LoadBalancerProperties properties;

    public GrayReactiveLoadBalancerClientFilter(LoadBalancerClientFactory clientFactory,
            LoadBalancerProperties properties) {
        this.clientFactory = clientFactory;
        this.properties = properties;
    }

    @Override
    public int getOrder() {
        return LOAD_BALANCER_CLIENT_FILTER_ORDER - 1;
    }

    @Override
    @SuppressWarnings("Duplicates")
    public Mono<Void> filter(ServerWebExchange exchange, GatewayFilterChain chain) {
        URI url = exchange.getAttribute(GATEWAY_REQUEST_URL_ATTR);
        String schemePrefix = exchange.getAttribute(GATEWAY_SCHEME_PREFIX_ATTR);
        if (url == null || (!"lb".equals(url.getScheme()) && !"lb".equals(schemePrefix))) {
            return chain.filter(exchange);
        }
        // preserve the original url
        addOriginalRequestUrl(exchange, url);

        if (log.isTraceEnabled()) {
            log.trace(ReactiveLoadBalancerClientFilter.class.getSimpleName() + " url before: " + url);
        }

        return choose(exchange).doOnNext(response -> {

            if (!response.hasServer()) {
                throw NotFoundException.create(properties.isUse404(), "Unable to find instance for " + url.getHost());
            }

            ServiceInstance retrievedInstance = response.getServer();

            URI uri = exchange.getRequest().getURI();

            // if the `lb:<scheme>` mechanism was used, use `<scheme>` as the default,
            // if the loadbalancer doesn't provide one.
            String overrideScheme = retrievedInstance.isSecure() ? "https" : "http";
            if (schemePrefix != null) {
                overrideScheme = url.getScheme();
            }

            DelegatingServiceInstance serviceInstance = new DelegatingServiceInstance(retrievedInstance,
                    overrideScheme);

            URI requestUrl = reconstructURI(serviceInstance, uri);

            if (log.isTraceEnabled()) {
                log.trace("LoadBalancerClientFilter url chosen: " + requestUrl);
            }
            exchange.getAttributes().put(GATEWAY_REQUEST_URL_ATTR, requestUrl);
        }).then(chain.filter(exchange));
    }

    protected URI reconstructURI(ServiceInstance serviceInstance, URI original) {
        return LoadBalancerUriTools.reconstructURI(serviceInstance, original);
    }

    @SuppressWarnings("deprecation")
    private Mono<Response<ServiceInstance>> choose(ServerWebExchange exchange) {
        URI uri = exchange.getAttribute(GATEWAY_REQUEST_URL_ATTR);
        ReactorLoadBalancer<ServiceInstance> loadBalancer = this.clientFactory.getInstance(uri.getHost(),
                ReactorServiceInstanceLoadBalancer.class);
        if (loadBalancer == null) {
            throw new NotFoundException("No loadbalancer available for " + uri.getHost());
        }
        return loadBalancer.choose(createRequest(exchange));
    }

    @SuppressWarnings("deprecation")
    private Request createRequest(ServerWebExchange exchange) {
        HttpHeaders headers = exchange.getRequest().getHeaders();
        return new DefaultRequest<>(headers);
    }

}

SPI注入

package cn.idea360.gateway.gray;

import cn.idea360.gateway.gray.filter.GrayReactiveLoadBalancerClientFilter;
import cn.idea360.gateway.gray.lb.NacosGrayLoadBalancer;
import cn.idea360.gateway.gray.properties.GrayProperties;
import cn.idea360.gateway.gray.rule.GrayRuleManage;
import org.springframework.boot.autoconfigure.condition.ConditionalOnBean;
import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.cloud.client.ServiceInstance;
import org.springframework.cloud.gateway.config.LoadBalancerProperties;
import org.springframework.cloud.loadbalancer.annotation.LoadBalancerClients;
import org.springframework.cloud.loadbalancer.core.ReactorLoadBalancer;
import org.springframework.cloud.loadbalancer.core.ServiceInstanceListSupplier;
import org.springframework.cloud.loadbalancer.support.LoadBalancerClientFactory;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.env.Environment;

/**
 * @author cuishiying
 * @date 2021-01-20
 */
@LoadBalancerClients(defaultConfiguration = GrayAutoConfiguration.class)
@Configuration
@EnableConfigurationProperties(GrayProperties.class)
public class GrayAutoConfiguration {

    @Bean
    @ConditionalOnBean({ LoadBalancerClientFactory.class })
    public GrayReactiveLoadBalancerClientFilter grayReactiveLoadBalancerClientFilter(
            LoadBalancerClientFactory clientFactory, LoadBalancerProperties properties) {
        return new GrayReactiveLoadBalancerClientFilter(clientFactory, properties);
    }

    @Bean
    @ConditionalOnMissingBean
    public GrayRuleManage grayRuleManage(GrayProperties grayProperties) {
        return new GrayRuleManage(grayProperties);
    }

    @Bean
    @ConditionalOnBean({ LoadBalancerClientFactory.class })
    public ReactorLoadBalancer<ServiceInstance> reactorServiceInstanceLoadBalancer(Environment environment,
            LoadBalancerClientFactory loadBalancerClientFactory, GrayRuleManage grayRuleManage) {
        String name = environment.getProperty(LoadBalancerClientFactory.PROPERTY_NAME);
        return new NacosGrayLoadBalancer(
                loadBalancerClientFactory.getLazyProvider(name, ServiceInstanceListSupplier.class), name,
                grayRuleManage);
    }

}

发布灰度服务

# 应用名称
spring.application.name=provider
# 应用服务 WEB 访问端口
server.port=9002

spring.cloud.nacos.config.server-addr=127.0.0.1:8848
spring.cloud.nacos.discovery.server-addr=127.0.0.1:8848
spring.cloud.nacos.discovery.metadata.version = gray

测试, curl -X GET -H "version:gray" -d '{"name": "admin"}' http://127.0.0.1:9000/provider/test/port 发现会永远路由到 9002

方案总结

网关灰度方案(API网关 -> Consumer)

假设目前已完成了服务间灰度方案, 在网关过滤器中解析token并将cid和uid放在Header中
网关服务注册到nacos
客户的请求进入API网关
API网关读取nacos配置中的版本V对应的公司列表，如果该请求头中cid参数命中, 则网关读取自己在元数据中的版本号V, 并根据版本号V查找对应版本的C。如果未命中代表请求非灰度请求，服务取反。(这一步实际开发中可能需要调整)
最终结果网关中配置v=2&cid=xxx的请求进入V2版本的C, 其他请求进入V1版本的C

Nginx和API网关配置相同的路由会发生什么?

一直有个疑问, 以前的项目中用 Nginx 作为网关, 假设需要改造引入 spring-cloud-gateway 网关, 在不影响其他项目组的情况下如何局部应用api网关呢? 今天测试了下Nginx配置文件的优先级。

1 2	Nginx中 `/api` 配置网关根路径, 网关中配置 `/service-a/echo` 去访问A服务的接口 Nginx中 `/api/service-a/echo` 去访问A服务

实际测试后发现无论Nginx配置文件中的顺序如何, 都会通过nginx去访问 /api/service-a/echo, 不会经过api网关。

最后

本文到此结束，感谢大家的阅读。欢迎大家关注公众号【当我遇上你】。

本文链接： https://idea360.cn/2021/07/10/spring-cloud-gateway-gray/

版权声明： 本博客所有文章除特别声明外，均采用 CC BY 4.0 CN协议许可协议。转载请注明出处！